source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-auxmm ` `kubectl create namespace ns-auxmm` namespace/ns-auxmm created create namespace ns-auxmm done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 143M 0 --:--:-- --:--:-- --:--:-- 143M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:postgresql set component version set component version:vanilla-postgresql set service versions:12.15.0,14.7.0,15.6.1138,15.7.0 set service versions sorted:12.15.0,14.7.0,15.6.1138,15.7.0 no cluster version found unsupported component definition set replicas first:2,12.15.0|2,14.7.0|2,15.6.1138|2,15.7.0 set replicas third:2,14.7.0 set replicas fourth:2,14.7.0 set minimum cmpv service version set minimum cmpv service version replicas:2,14.7.0 REPORT_COUNT:1 CLUSTER_TOPOLOGY: set cluster topology: vanilla-postgresql LIMIT_CPU:0.5 LIMIT_MEMORY:0.5 storage size: 3 No resources found in ns-auxmm namespace. termination_policy:Halt create 2 replica Halt vanilla-postgresql cluster check cluster definition check component definition set component definition by component version check cmpd by labels set component definition1: vanilla-postgresql-14 by component version:vanilla-postgresql apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: vanilla-kzgbtp namespace: ns-auxmm spec: clusterDefinitionRef: vanilla-postgresql topology: vanilla-postgresql terminationPolicy: Halt componentSpecs: - name: postgresql serviceVersion: 14.7.0 replicas: 2 resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 3Gi `kubectl apply -f test_create_vanilla-kzgbtp.yaml` cluster.apps.kubeblocks.io/vanilla-kzgbtp created apply test_create_vanilla-kzgbtp.yaml Success `rm -rf test_create_vanilla-kzgbtp.yaml` check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Jun 19,2025 18:15 UTC+0800 clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status: cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:15 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:15 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=vanilla-kzgbtp` `kubectl get secrets vanilla-kzgbtp-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets vanilla-kzgbtp-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets vanilla-kzgbtp-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:2s43NuhF79;DB_PORT:5432;DB_DATABASE: check pod vanilla-kzgbtp-postgresql-0 container_name postgresql exist password 2s43NuhF79 check pod vanilla-kzgbtp-postgresql-0 container_name lorry exist password 2s43NuhF79 check pod vanilla-kzgbtp-postgresql-0 container_name config-manager exist password 2s43NuhF79 No container logs contain secret password. describe cluster `kbcli cluster describe vanilla-kzgbtp --namespace ns-auxmm ` Name: vanilla-kzgbtp Created Time: Jun 19,2025 18:15 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-auxmm vanilla-postgresql Running Halt Endpoints: COMPONENT MODE INTERNAL EXTERNAL postgresql ReadWrite vanilla-kzgbtp-postgresql.ns-auxmm.svc.cluster.local:5432 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME postgresql vanilla-kzgbtp-postgresql-0 primary Running us-west-2a ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:15 UTC+0800 postgresql vanilla-kzgbtp-postgresql-1 secondary Running us-west-2a ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:15 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS postgresql false 500m / 500m 512Mi / 512Mi data:3Gi kb-default-sc Images: COMPONENT TYPE IMAGE postgresql docker.io/apecloud/postgres:14.7 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-auxmm vanilla-kzgbtp `kbcli cluster label vanilla-kzgbtp app.kubernetes.io/instance- --namespace ns-auxmm ` label "app.kubernetes.io/instance" not found. `kbcli cluster label vanilla-kzgbtp app.kubernetes.io/instance=vanilla-kzgbtp --namespace ns-auxmm ` `kbcli cluster label vanilla-kzgbtp --list --namespace ns-auxmm ` NAME NAMESPACE LABELS vanilla-kzgbtp ns-auxmm app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= label cluster app.kubernetes.io/instance=vanilla-kzgbtp Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=vanilla-kzgbtp --namespace ns-auxmm ` `kbcli cluster label vanilla-kzgbtp --list --namespace ns-auxmm ` NAME NAMESPACE LABELS vanilla-kzgbtp ns-auxmm app.kubernetes.io/instance=vanilla-kzgbtp case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test1 Success `kbcli cluster label vanilla-kzgbtp case.name=kbcli.test2 --overwrite --namespace ns-auxmm ` `kbcli cluster label vanilla-kzgbtp --list --namespace ns-auxmm ` NAME NAMESPACE LABELS vanilla-kzgbtp ns-auxmm app.kubernetes.io/instance=vanilla-kzgbtp case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test2 Success `kbcli cluster label vanilla-kzgbtp case.name- --namespace ns-auxmm ` `kbcli cluster label vanilla-kzgbtp --list --namespace ns-auxmm ` NAME NAMESPACE LABELS vanilla-kzgbtp ns-auxmm app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= delete cluster label case.name Success cluster connect `echo 'select 1' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance vanilla-kzgbtp-postgresql-0: out of vanilla-kzgbtp-postgresql-0(primary), vanilla-kzgbtp-postgresql-1(secondary) ?column? ---------- 1 (1 row) connect cluster Success cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose vanilla-kzgbtp --auto-approve --force=true --type vpc --enable true --components postgresql --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-expose-dps88 created successfully, you can view the progress: kbcli cluster describe-ops vanilla-kzgbtp-expose-dps88 -n ns-auxmm check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-expose-dps88 ns-auxmm Expose vanilla-kzgbtp postgresql Running 0/1 Jun 19,2025 18:17 UTC+0800 check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:15 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:15 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-expose-dps88 ns-auxmm Expose vanilla-kzgbtp postgresql Succeed 1/1 Jun 19,2025 18:17 UTC+0800 check ops status done ops_status:vanilla-kzgbtp-expose-dps88 ns-auxmm Expose vanilla-kzgbtp postgresql Succeed 1/1 Jun 19,2025 18:17 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-kzgbtp-expose-dps88 --namespace ns-auxmm ` opsrequest.apps.kubeblocks.io/vanilla-kzgbtp-expose-dps88 patched `kbcli cluster delete-ops --name vanilla-kzgbtp-expose-dps88 --force --auto-approve --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-expose-dps88 deleted test failover kill1 check node drain check node drain success `kill 1` Defaulted container "postgresql" out of: postgresql, lorry, config-manager, init-syncer (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:15 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:15 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-1;secondary: vanilla-kzgbtp-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done check failover pod name failover pod name:vanilla-kzgbtp-postgresql-1 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=vanilla-kzgbtp` `kubectl get secrets vanilla-kzgbtp-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets vanilla-kzgbtp-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets vanilla-kzgbtp-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:2s43NuhF79;DB_PORT:5432;DB_DATABASE: `create database benchtest;` Unable to use a TTY - input is not a terminal or the right kind of file NOTICE: database "benchtest" does not exist, skipping return msg:Connect to instance vanilla-kzgbtp-postgresql-1: out of vanilla-kzgbtp-postgresql-1(primary), vanilla-kzgbtp-postgresql-0(secondary) DROP DATABASE CREATE DATABASE apiVersion: v1 kind: Pod metadata: name: benchtest-vanilla-kzgbtp namespace: ns-auxmm spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:pgsql,host:vanilla-kzgbtp-postgresql.ns-auxmm.svc.cluster.local,user:postgres,password:2s43NuhF79,port:5432,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-vanilla-kzgbtp.yaml` pod/benchtest-vanilla-kzgbtp created apply benchtest-vanilla-kzgbtp.yaml Success `rm -rf benchtest-vanilla-kzgbtp.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 18s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 24s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 1/1 Running 0 36s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 1/1 Running 0 42s check pod benchtest-vanilla-kzgbtp status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 Completed 0 48s `kubectl logs benchtest-vanilla-kzgbtp --tail 30 --namespace ns-auxmm ` [ 7s ] thds: 4 tps: 163.01 qps: 3251.30 (r/w/o: 2273.21/652.06/326.03) lat (ms,99%): 78.60 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 180.98 qps: 3645.59 (r/w/o: 2554.71/722.92/367.96) lat (ms,99%): 78.60 err/s: 1.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 170.56 qps: 3412.11 (r/w/o: 2387.78/681.22/343.11) lat (ms,99%): 78.60 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 171.46 qps: 3440.32 (r/w/o: 2408.52/688.87/342.93) lat (ms,99%): 78.60 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 22904 write: 6523 other: 3283 total: 32710 transactions: 1631 (161.89 per sec.) queries: 32710 (3246.67 per sec.) ignored errors: 5 (0.50 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0733s total number of events: 1631 Latency (ms): min: 5.28 avg: 24.65 max: 91.95 99th percentile: 84.47 sum: 40206.12 Threads fairness: events (avg/stddev): 407.7500/6.18 execution time (avg/stddev): 10.0515/0.03 `kubectl delete pod benchtest-vanilla-kzgbtp --force --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-vanilla-kzgbtp" force deleted No resources found in ns-auxmm namespace. `kubectl get secrets -l app.kubernetes.io/instance=vanilla-kzgbtp` `kubectl get secrets vanilla-kzgbtp-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets vanilla-kzgbtp-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets vanilla-kzgbtp-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:2s43NuhF79;DB_PORT:5432;DB_DATABASE: `create database benchtest;` Unable to use a TTY - input is not a terminal or the right kind of file return msg:Connect to instance vanilla-kzgbtp-postgresql-1: out of vanilla-kzgbtp-postgresql-1(primary), vanilla-kzgbtp-postgresql-0(secondary) DROP DATABASE CREATE DATABASE apiVersion: v1 kind: Pod metadata: name: benchtest-vanilla-kzgbtp namespace: ns-auxmm spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:pgsql,host:a96e415ecd3944eed869eb022ab718c2-a259841339e614d3.elb.us-west-2.amazonaws.com,user:postgres,password:2s43NuhF79,port:5432,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-vanilla-kzgbtp.yaml` pod/benchtest-vanilla-kzgbtp created apply benchtest-vanilla-kzgbtp.yaml Success `rm -rf benchtest-vanilla-kzgbtp.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 18s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 24s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 30s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 36s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 ContainerCreating 0 42s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 1/1 Running 0 49s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 1/1 Running 0 55s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 1/1 Running 0 61s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 1/1 Running 0 67s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 1/1 Running 0 73s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 1/1 Running 0 79s check pod benchtest-vanilla-kzgbtp status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-kzgbtp 0/1 Completed 0 85s `kubectl logs benchtest-vanilla-kzgbtp --tail 30 --namespace ns-auxmm ` [ 7s ] thds: 4 tps: 154.96 qps: 3110.16 (r/w/o: 2181.41/614.83/313.91) lat (ms,99%): 78.60 err/s: 1.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 155.02 qps: 3087.34 (r/w/o: 2159.24/617.07/311.03) lat (ms,99%): 73.13 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 165.02 qps: 3309.46 (r/w/o: 2314.32/663.09/332.05) lat (ms,99%): 74.46 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 163.99 qps: 3310.86 (r/w/o: 2323.90/653.97/332.99) lat (ms,99%): 75.82 err/s: 2.00 reconn/s: 0.00 SQL statistics: queries performed: read: 19894 write: 5657 other: 2851 total: 28402 transactions: 1413 (141.15 per sec.) queries: 28402 (2837.17 per sec.) ignored errors: 8 (0.80 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0090s total number of events: 1413 Latency (ms): min: 7.98 avg: 28.32 max: 1029.85 99th percentile: 78.60 sum: 40018.21 Threads fairness: events (avg/stddev): 353.2500/18.98 execution time (avg/stddev): 10.0046/0.00 `kubectl delete pod benchtest-vanilla-kzgbtp --force --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-vanilla-kzgbtp" force deleted cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale vanilla-kzgbtp --auto-approve --force=true --components postgresql --cpu 600m --memory 0.6Gi --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-verticalscaling-rndnv created successfully, you can view the progress: kbcli cluster describe-ops vanilla-kzgbtp-verticalscaling-rndnv -n ns-auxmm check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-verticalscaling-rndnv ns-auxmm VerticalScaling vanilla-kzgbtp postgresql Running 0/2 Jun 19,2025 18:22 UTC+0800 check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:22 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:22 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-verticalscaling-rndnv ns-auxmm VerticalScaling vanilla-kzgbtp postgresql Succeed 2/2 Jun 19,2025 18:22 UTC+0800 check ops status done ops_status:vanilla-kzgbtp-verticalscaling-rndnv ns-auxmm VerticalScaling vanilla-kzgbtp postgresql Succeed 2/2 Jun 19,2025 18:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-kzgbtp-verticalscaling-rndnv --namespace ns-auxmm ` opsrequest.apps.kubeblocks.io/vanilla-kzgbtp-verticalscaling-rndnv patched `kbcli cluster delete-ops --name vanilla-kzgbtp-verticalscaling-rndnv --force --auto-approve --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-verticalscaling-rndnv deleted test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-vanilla-kzgbtp created apply test-chaos-mesh-networkdelay-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkdelay-vanilla-kzgbtp.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:22 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:22 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-0 failover networkdelay Success test failover delete pod:vanilla-kzgbtp-postgresql-0 `kubectl delete pod vanilla-kzgbtp-postgresql-0 --namespace ns-auxmm ` pod "vanilla-kzgbtp-postgresql-0" deleted check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:26 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:22 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-1;secondary: vanilla-kzgbtp-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done check failover pod name failover pod name:vanilla-kzgbtp-postgresql-1 failover Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-1 action: partition mode: all target: mode: all selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-vanilla-kzgbtp created apply test-chaos-mesh-networkpartition-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkpartition-vanilla-kzgbtp.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:26 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:22 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-1;secondary: vanilla-kzgbtp-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-1 failover networkpartition Success test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-vanilla-kzgbtp" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-vanilla-kzgbtp.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-vanilla-kzgbtp created apply test-chaos-mesh-fullcpu-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-fullcpu-vanilla-kzgbtp.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:26 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:22 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-1;secondary: vanilla-kzgbtp-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-vanilla-kzgbtp" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-1 failover fullcpu Success skip Reconfiguring: skip Reconfiguring: cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop vanilla-kzgbtp --auto-approve --force=true --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-stop-bf5gh created successfully, you can view the progress: kbcli cluster describe-ops vanilla-kzgbtp-stop-bf5gh -n ns-auxmm check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-stop-bf5gh ns-auxmm Stop vanilla-kzgbtp postgresql Running 0/2 Jun 19,2025 18:31 UTC+0800 check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Stopped Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-stop-bf5gh ns-auxmm Stop vanilla-kzgbtp postgresql Succeed 2/2 Jun 19,2025 18:31 UTC+0800 check ops status done ops_status:vanilla-kzgbtp-stop-bf5gh ns-auxmm Stop vanilla-kzgbtp postgresql Succeed 2/2 Jun 19,2025 18:31 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-kzgbtp-stop-bf5gh --namespace ns-auxmm ` opsrequest.apps.kubeblocks.io/vanilla-kzgbtp-stop-bf5gh patched `kbcli cluster delete-ops --name vanilla-kzgbtp-stop-bf5gh --force --auto-approve --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-stop-bf5gh deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start vanilla-kzgbtp --force=true --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-start-tkjds created successfully, you can view the progress: kbcli cluster describe-ops vanilla-kzgbtp-start-tkjds -n ns-auxmm check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-start-tkjds ns-auxmm Start vanilla-kzgbtp postgresql Running 0/2 Jun 19,2025 18:31 UTC+0800 check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:31 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-start-tkjds ns-auxmm Start vanilla-kzgbtp postgresql Succeed 2/2 Jun 19,2025 18:31 UTC+0800 check ops status done ops_status:vanilla-kzgbtp-start-tkjds ns-auxmm Start vanilla-kzgbtp postgresql Succeed 2/2 Jun 19,2025 18:31 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-kzgbtp-start-tkjds --namespace ns-auxmm ` opsrequest.apps.kubeblocks.io/vanilla-kzgbtp-start-tkjds patched `kbcli cluster delete-ops --name vanilla-kzgbtp-start-tkjds --force --auto-approve --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-start-tkjds deleted test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-vanilla-kzgbtp" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-vanilla-kzgbtp.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-vanilla-kzgbtp created apply test-chaos-mesh-dnsrandom-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-dnsrandom-vanilla-kzgbtp.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:31 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-vanilla-kzgbtp" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-0 failover dnsrandom Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-kzgbtp" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-vanilla-kzgbtp.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-vanilla-kzgbtp created apply test-chaos-mesh-oom-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-oom-vanilla-kzgbtp.yaml` check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:31 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-kzgbtp" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-vanilla-kzgbtp patched check failover pod name failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-kzgbtp" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-kzgbtp" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-vanilla-kzgbtp.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-vanilla-kzgbtp created apply test-chaos-mesh-oom-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-oom-vanilla-kzgbtp.yaml` check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:31 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-1;secondary: vanilla-kzgbtp-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-kzgbtp" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-kzgbtp" not found check failover pod name:vanilla-kzgbtp-postgresql-1 failover oom Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-vanilla-kzgbtp" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-1 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-vanilla-kzgbtp.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-vanilla-kzgbtp created apply test-chaos-mesh-dnserror-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-dnserror-vanilla-kzgbtp.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-11.us-west-2.compute.internal/172.31.2.11 Jun 19,2025 18:31 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-1;secondary: vanilla-kzgbtp-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-vanilla-kzgbtp" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-1 failover dnserror Success 11 test failover drainnode check node drain check node drain success kubectl get pod vanilla-kzgbtp-postgresql-1 --namespace ns-auxmm -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-2-11.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod vanilla-kzgbtp-postgresql-0 --namespace ns-auxmm -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-6-29.us-west-2.compute.internal success kubectl drain ip-172-31-2-11.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-2-11.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-594pq, kb-devqu/kb-addon-apecloud-otel-collector-jcnfs, kube-system/aws-node-4ntd7, kube-system/ebs-csi-node-lhh4g, kube-system/kube-proxy-lglc6 evicting pod ns-auxmm/vanilla-kzgbtp-postgresql-1 evicting pod kb-devqu/kb-addon-snapshot-controller-8585f8b95f-cb9t9 pod/kb-addon-snapshot-controller-8585f8b95f-cb9t9 evicted pod/vanilla-kzgbtp-postgresql-1 evicted node/ip-172-31-2-11.us-west-2.compute.internal drained kubectl uncordon ip-172-31-2-11.us-west-2.compute.internal node/ip-172-31-2-11.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done check failover pod name failover pod name:vanilla-kzgbtp-postgresql-0 failover drainnode Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-vanilla-kzgbtp" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-vanilla-kzgbtp.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-vanilla-kzgbtp created apply test-chaos-mesh-podfailure-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-podfailure-vanilla-kzgbtp.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-1;secondary: vanilla-kzgbtp-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-vanilla-kzgbtp" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-1 failover podfailure Success test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-1 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-vanilla-kzgbtp created apply test-chaos-mesh-networklossover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networklossover-vanilla-kzgbtp.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-0 failover networklossover Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-vanilla-kzgbtp" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-vanilla-kzgbtp.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-vanilla-kzgbtp created apply test-chaos-mesh-timeoffset-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-timeoffset-vanilla-kzgbtp.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-vanilla-kzgbtp" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-0 failover timeoffset Success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp created apply test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp created apply test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp created apply test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp created apply test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp created apply test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp created apply test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-kzgbtp" not found check failover pod name timeout `kubectl get pvc -l app.kubernetes.io/instance=vanilla-kzgbtp,apps.kubeblocks.io/component-name=postgresql,apps.kubeblocks.io/vct-name=data --namespace ns-auxmm ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in vanilla-kzgbtp namespace. `kbcli cluster volume-expand vanilla-kzgbtp --auto-approve --force=true --components postgresql --volume-claim-templates data --storage 6Gi --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-volumeexpansion-mwx58 created successfully, you can view the progress: kbcli cluster describe-ops vanilla-kzgbtp-volumeexpansion-mwx58 -n ns-auxmm check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-volumeexpansion-mwx58 ns-auxmm VolumeExpansion vanilla-kzgbtp postgresql Running 0/2 Jun 19,2025 19:00 UTC+0800 check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done No resources found in vanilla-kzgbtp namespace. check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-volumeexpansion-mwx58 ns-auxmm VolumeExpansion vanilla-kzgbtp postgresql Succeed 2/2 Jun 19,2025 19:00 UTC+0800 check ops status done ops_status:vanilla-kzgbtp-volumeexpansion-mwx58 ns-auxmm VolumeExpansion vanilla-kzgbtp postgresql Succeed 2/2 Jun 19,2025 19:00 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-kzgbtp-volumeexpansion-mwx58 --namespace ns-auxmm ` opsrequest.apps.kubeblocks.io/vanilla-kzgbtp-volumeexpansion-mwx58 patched `kbcli cluster delete-ops --name vanilla-kzgbtp-volumeexpansion-mwx58 --force --auto-approve --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-volumeexpansion-mwx58 deleted test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-vanilla-kzgbtp created apply test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-vanilla-kzgbtp created apply test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-vanilla-kzgbtp patched failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-vanilla-kzgbtp created apply test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-vanilla-kzgbtp created apply test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-vanilla-kzgbtp created apply test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found failover pod name:vanilla-kzgbtp-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-vanilla-kzgbtp created apply test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-vanilla-kzgbtp.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 18:38 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-kzgbtp-postgresql-0;secondary: vanilla-kzgbtp-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-kzgbtp --namespace ns-auxmm` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-kzgbtp" not found check failover pod name timeout cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart vanilla-kzgbtp --auto-approve --force=true --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-restart-xpvgz created successfully, you can view the progress: kbcli cluster describe-ops vanilla-kzgbtp-restart-xpvgz -n ns-auxmm check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 0/2 Jun 19,2025 19:16 UTC+0800 check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed [Error] check cluster status timeout --------------------------------------get cluster vanilla-kzgbtp yaml-------------------------------------- `kubectl get cluster vanilla-kzgbtp -o yaml --namespace ns-auxmm ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"vanilla-kzgbtp-restart-xpvgz","type":"Restart"***]' kubeblocks.io/reconcile: "2025-06-19T11:21:34.27653533Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"vanilla-kzgbtp","namespace":"ns-auxmm"***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","componentSpecs":[***"name":"postgresql","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"14.7.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"3Gi"***,"storageClassName":null***]***],"terminationPolicy":"Halt","topology":"vanilla-postgresql"*** creationTimestamp: "2025-06-19T10:15:26Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 9 labels: app.kubernetes.io/instance: vanilla-kzgbtp clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" name: vanilla-kzgbtp namespace: ns-auxmm resourceVersion: "73276" uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 spec: clusterDefinitionRef: vanilla-postgresql componentSpecs: - componentDef: vanilla-postgresql-14 name: postgresql replicas: 2 resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m serviceVersion: 14.7.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 6Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: postgresql name: postgresql-vpc roleSelector: primary serviceName: postgresql-vpc spec: ports: - name: tcp-postgresql nodePort: 30267 port: 5432 protocol: TCP targetPort: tcp-postgresql type: LoadBalancer storage: size: "0" terminationPolicy: Halt topology: vanilla-postgresql status: clusterDefGeneration: 1 components: postgresql: message: InstanceSet/vanilla-kzgbtp-postgresql: '["vanilla-kzgbtp-postgresql-1"]' phase: Failed podsReady: false podsReadyTime: "2025-06-19T11:15:52Z" conditions: - lastTransitionTime: "2025-06-19T10:15:26Z" message: 'The operator has started the provisioning of Cluster: vanilla-kzgbtp' observedGeneration: 9 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:17:01Z" message: Successfully applied for resources observedGeneration: 9 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 9 phase: Failed ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster vanilla-kzgbtp-------------------------------------- `kubectl describe cluster vanilla-kzgbtp --namespace ns-auxmm ` Name: vanilla-kzgbtp Namespace: ns-auxmm Labels: app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/ops-request: [***"name":"vanilla-kzgbtp-restart-xpvgz","type":"Restart"***] kubeblocks.io/reconcile: 2025-06-19T11:21:34.27653533Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:26Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 9 Resource Version: 73276 UID: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 Spec: Cluster Definition Ref: vanilla-postgresql Component Specs: Component Def: vanilla-postgresql-14 Name: postgresql Replicas: 2 Resources: Limits: Cpu: 600m Memory: 644245094400m Requests: Cpu: 600m Memory: 644245094400m Service Version: 14.7.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 6Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: postgresql Name: postgresql-vpc Role Selector: primary Service Name: postgresql-vpc Spec: Ports: Name: tcp-postgresql Node Port: 30267 Port: 5432 Protocol: TCP Target Port: tcp-postgresql Type: LoadBalancer Storage: Size: 0 Termination Policy: Halt Topology: vanilla-postgresql Status: Cluster Def Generation: 1 Components: Postgresql: Message: InstanceSet/vanilla-kzgbtp-postgresql: ["vanilla-kzgbtp-postgresql-1"] Phase: Failed Pods Ready: false Pods Ready Time: 2025-06-19T11:15:52Z Conditions: Last Transition Time: 2025-06-19T10:15:26Z Message: The operator has started the provisioning of Cluster: vanilla-kzgbtp Observed Generation: 9 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:17:01Z Message: Successfully applied for resources Observed Generation: 9 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:16:07Z Message: pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:16:07Z Message: pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 9 Phase: Failed Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal HorizontalScale 53m component-controller start horizontal scale component postgresql of cluster vanilla-kzgbtp from 0 to 2 Normal ComponentPhaseTransition 49m cluster-controller component is Failed Warning BackOff 45m (x6 over 45m) event-controller Pod vanilla-kzgbtp-postgresql-0: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-0_ns-auxmm(f3b3a75d-ecb7-4978-baeb-a7597f9d8a14) Warning Failed 44m event-controller Pod vanilla-kzgbtp-postgresql-0: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "lorry": executable file not found in $PATH: unknown Normal ComponentPhaseTransition 37m (x14 over 67m) cluster-controller component is Updating Normal PreCheckSucceed 24m (x9 over 69m) cluster-controller The operator has started the provisioning of Cluster: vanilla-kzgbtp Warning Unhealthy 9m33s (x7 over 37m) event-controller Pod vanilla-kzgbtp-postgresql-0: Readiness probe failed: Get "http://172.31.11.240:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal ComponentPhaseTransition 9m9s (x35 over 68m) cluster-controller component is Running Warning BackOff 3m56s (x19 over 7m59s) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql CrashLoopBackOff us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 19:16 UTC+0800 pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:Error pod_status:Error pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:Error pod_status:Error pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff [Error] check pod status timeout --------------------------------------get cluster vanilla-kzgbtp yaml-------------------------------------- `kubectl get cluster vanilla-kzgbtp -o yaml --namespace ns-auxmm ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"vanilla-kzgbtp-restart-xpvgz","type":"Restart"***]' kubeblocks.io/reconcile: "2025-06-19T11:31:32.273461683Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"vanilla-kzgbtp","namespace":"ns-auxmm"***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","componentSpecs":[***"name":"postgresql","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"14.7.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"3Gi"***,"storageClassName":null***]***],"terminationPolicy":"Halt","topology":"vanilla-postgresql"*** creationTimestamp: "2025-06-19T10:15:26Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 9 labels: app.kubernetes.io/instance: vanilla-kzgbtp clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" name: vanilla-kzgbtp namespace: ns-auxmm resourceVersion: "81687" uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 spec: clusterDefinitionRef: vanilla-postgresql componentSpecs: - componentDef: vanilla-postgresql-14 name: postgresql replicas: 2 resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m serviceVersion: 14.7.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 6Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: postgresql name: postgresql-vpc roleSelector: primary serviceName: postgresql-vpc spec: ports: - name: tcp-postgresql nodePort: 30267 port: 5432 protocol: TCP targetPort: tcp-postgresql type: LoadBalancer storage: size: "0" terminationPolicy: Halt topology: vanilla-postgresql status: clusterDefGeneration: 1 components: postgresql: message: InstanceSet/vanilla-kzgbtp-postgresql: '["vanilla-kzgbtp-postgresql-1"]' phase: Failed podsReady: false podsReadyTime: "2025-06-19T11:15:52Z" conditions: - lastTransitionTime: "2025-06-19T10:15:26Z" message: 'The operator has started the provisioning of Cluster: vanilla-kzgbtp' observedGeneration: 9 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:17:01Z" message: Successfully applied for resources observedGeneration: 9 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 9 phase: Failed ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster vanilla-kzgbtp-------------------------------------- `kubectl describe cluster vanilla-kzgbtp --namespace ns-auxmm ` Name: vanilla-kzgbtp Namespace: ns-auxmm Labels: app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/ops-request: [***"name":"vanilla-kzgbtp-restart-xpvgz","type":"Restart"***] kubeblocks.io/reconcile: 2025-06-19T11:31:32.273461683Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:26Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 9 Resource Version: 81687 UID: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 Spec: Cluster Definition Ref: vanilla-postgresql Component Specs: Component Def: vanilla-postgresql-14 Name: postgresql Replicas: 2 Resources: Limits: Cpu: 600m Memory: 644245094400m Requests: Cpu: 600m Memory: 644245094400m Service Version: 14.7.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 6Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: postgresql Name: postgresql-vpc Role Selector: primary Service Name: postgresql-vpc Spec: Ports: Name: tcp-postgresql Node Port: 30267 Port: 5432 Protocol: TCP Target Port: tcp-postgresql Type: LoadBalancer Storage: Size: 0 Termination Policy: Halt Topology: vanilla-postgresql Status: Cluster Def Generation: 1 Components: Postgresql: Message: InstanceSet/vanilla-kzgbtp-postgresql: ["vanilla-kzgbtp-postgresql-1"] Phase: Failed Pods Ready: false Pods Ready Time: 2025-06-19T11:15:52Z Conditions: Last Transition Time: 2025-06-19T10:15:26Z Message: The operator has started the provisioning of Cluster: vanilla-kzgbtp Observed Generation: 9 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:17:01Z Message: Successfully applied for resources Observed Generation: 9 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:16:07Z Message: pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:16:07Z Message: pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 9 Phase: Failed Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 54m (x6 over 54m) event-controller Pod vanilla-kzgbtp-postgresql-0: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-0_ns-auxmm(f3b3a75d-ecb7-4978-baeb-a7597f9d8a14) Warning Failed 54m event-controller Pod vanilla-kzgbtp-postgresql-0: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "lorry": executable file not found in $PATH: unknown Normal PreCheckSucceed 33m (x9 over 79m) cluster-controller The operator has started the provisioning of Cluster: vanilla-kzgbtp Warning Unhealthy 19m (x7 over 47m) event-controller Pod vanilla-kzgbtp-postgresql-0: Readiness probe failed: Get "http://172.31.11.240:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal ComponentPhaseTransition 18m (x35 over 78m) cluster-controller component is Running Normal ComponentPhaseTransition 7m4s (x8 over 59m) cluster-controller component is Failed Warning BackOff 3m3s (x22 over 17m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) Normal ComponentPhaseTransition 2m10s (x44 over 77m) cluster-controller component is Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------get pod vanilla-kzgbtp-postgresql-0 vanilla-kzgbtp-postgresql-1 yaml-------------------------------------- `kubectl get pod vanilla-kzgbtp-postgresql-0 -o yaml --namespace ns-auxmm ` apiVersion: v1 kind: Pod metadata: annotations: apps.kubeblocks.io/component-replicas: "2" apps.kubeblocks.io/last-role-snapshot-version: "1750329814096971" creationTimestamp: "2025-06-19T10:31:40Z" labels: app.kubernetes.io/component: vanilla-postgresql-14 app.kubernetes.io/instance: vanilla-kzgbtp app.kubernetes.io/managed-by: kubeblocks app.kubernetes.io/name: vanilla-postgresql-14 app.kubernetes.io/version: vanilla-postgresql-14 apps.kubeblocks.io/cluster-uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 apps.kubeblocks.io/component-name: postgresql apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" componentdefinition.kubeblocks.io/name: vanilla-postgresql-14 controller-revision-hash: 6c8d9d7bf5 kubeblocks.io/role: primary workloads.kubeblocks.io/access-mode: ReadWrite workloads.kubeblocks.io/instance: vanilla-kzgbtp-postgresql workloads.kubeblocks.io/managed-by: InstanceSet name: vanilla-kzgbtp-postgresql-0 namespace: ns-auxmm ownerReferences: - apiVersion: workloads.kubeblocks.io/v1alpha1 blockOwnerDeletion: true controller: true kind: InstanceSet name: vanilla-kzgbtp-postgresql uid: 7be55aad-9aa7-48fb-b705-0b349c86f2aa resourceVersion: "69281" uid: f3b3a75d-ecb7-4978-baeb-a7597f9d8a14 spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: kb-data operator: In values: - "true" weight: 100 containers: - command: - /kubeblocks/syncer - --port - "3601" - -- - docker-entrypoint.sh - --config-file=/postgresql/volume_data/conf/postgresql.conf - --hba_file=/postgresql/volume_data/conf/pg_hba.conf env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: ALLOW_NOSSL value: "true" - name: POSTGRESQL_PORT_NUMBER value: "5432" - name: PGDATA value: /postgresql/volume_data/data - name: PGCONF value: /postgresql/volume_data/conf - name: POSTGRESQL_MOUNTED_CONF_DIR value: /postgresql/mount_conf - name: PGUSER value: $(POSTGRES_USER) - name: PGPASSWORD value: $(POSTGRES_PASSWORD) - name: KB_ENGINE_TYPE value: vanilla-postgresql envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/postgres:14.7 imagePullPolicy: IfNotPresent name: postgresql ports: - containerPort: 5432 name: tcp-postgresql protocol: TCP resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m securityContext: runAsUser: 0 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /postgresql/volume_data name: data - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /kb-scripts name: scripts - mountPath: /kubeblocks name: kubeblocks - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true - command: - lorry - --port - "3501" - --grpcport - "50001" env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: KB_RSM_ROLE_PROBE_PERIOD value: "1" - name: KB_BUILTIN_HANDLER value: postgresql - name: KB_SERVICE_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_SERVICE_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_SERVICE_PORT value: "5432" - name: KB_DATA_PATH value: /postgresql/volume_data - name: KB_RSM_ACTION_SVC_LIST value: "null" - name: KB_RSM_ROLE_UPDATE_MECHANISM value: DirectAPIServerEventUpdate - name: KB_RSM_ROLE_PROBE_TIMEOUT value: "1" - name: KB_CLUSTER_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.labels['app.kubernetes.io/instance'] - name: KB_COMP_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.labels['apps.kubeblocks.io/component-name'] - name: KB_SERVICE_CHARACTER_TYPE value: postgresql envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/kubeblocks-tools:0.9.4 imagePullPolicy: IfNotPresent name: lorry ports: - containerPort: 3501 name: lorry-http-port protocol: TCP - containerPort: 50001 name: lorry-grpc-port protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /v1.0/checkrole port: 3501 scheme: HTTP periodSeconds: 1 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" startupProbe: failureThreshold: 3 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 3501 timeoutSeconds: 1 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/volume_data name: data - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true - args: - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) - /kb_tools/reloader - --log-level - info - --operator-update-enable - --tcp - "9901" - --config - /opt/config-manager/config-manager.yaml command: - env env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: CONFIG_MANAGER_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: TOOLS_PATH value: /opt/kb-tools/reload/vanilla-postgresql-configuration:/opt/config-manager:/kb_tools envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/postgres:14.7 imagePullPolicy: IfNotPresent name: config-manager ports: - containerPort: 9901 name: config-manager protocol: TCP resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostname: vanilla-kzgbtp-postgresql-0 initContainers: - command: - sh - -c - cp -r /bin/syncer /kubeblocks/ env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false image: docker.io/apecloud/syncer:0.3.6 imagePullPolicy: IfNotPresent name: init-syncer resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /kubeblocks name: kubeblocks - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true - command: - cp - /bin/reloader - /kb_tools env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false image: docker.io/apecloud/kubeblocks-tools:0.9.4 imagePullPolicy: IfNotPresent name: install-config-manager-tool resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true nodeName: ip-172-31-6-29.us-west-2.compute.internal preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 103 runAsGroup: 103 runAsUser: 0 serviceAccount: kb-vanilla-kzgbtp serviceAccountName: kb-vanilla-kzgbtp subdomain: vanilla-kzgbtp-postgresql-headless terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule key: kb-data operator: Equal value: "true" - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - emptyDir: medium: Memory sizeLimit: 644245094400m name: dshm - configMap: defaultMode: 292 name: vanilla-kzgbtp-postgresql-vanilla-postgresql-configuration name: postgresql-config - configMap: defaultMode: 365 name: vanilla-kzgbtp-postgresql-vanilla-postgresql-scripts name: scripts - configMap: defaultMode: 493 name: sidecar-vanilla-postgresql-reload-tools-script-vanilla-kzgbtp name: cm-script-vanilla-postgresql-configuration - configMap: defaultMode: 493 name: sidecar-vanilla-kzgbtp-postgresql-config-manager-config name: config-manager-config - emptyDir: *** name: kb-tools - name: data persistentVolumeClaim: claimName: data-vanilla-kzgbtp-postgresql-0 - emptyDir: *** name: kubeblocks - name: kube-api-access-lfdrn projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2025-06-19T10:31:45Z" status: "True" type: PodReadyToStartContainers - lastProbeTime: null lastTransitionTime: "2025-06-19T10:31:46Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2025-06-19T11:15:51Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2025-06-19T11:15:51Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2025-06-19T10:31:40Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://4a559d5430038eadb6ec4f2b6654901ec74628489218239605b3f1d1ae551206 image: docker.io/apecloud/postgres:14.7 imageID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c lastState: terminated: containerID: containerd://c4ff4251b5dc1f845c9cc8eacab8d2f149fa4c757d26c6a9c3db59a4e75a900f exitCode: 128 finishedAt: "2025-06-19T10:41:16Z" message: 'failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "env": executable file not found in $PATH: unknown' reason: StartError startedAt: "1970-01-01T00:00:00Z" name: config-manager ready: true restartCount: 6 started: true state: running: startedAt: "2025-06-19T10:41:37Z" volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled - containerID: containerd://fa38d6cf08bb686721e3f36bc4e802c5b83c45ea15d8c4b62134e4f1153a1a6d image: docker.io/apecloud/kubeblocks-tools:0.9.4 imageID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 lastState: terminated: containerID: containerd://e5fb6dd866135c11e8bf4dd6c9f1ac111a74578636f1ee6eb98f56360c34a7fa exitCode: 128 finishedAt: "2025-06-19T10:41:15Z" message: 'failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "lorry": executable file not found in $PATH: unknown' reason: StartError startedAt: "1970-01-01T00:00:00Z" name: lorry ready: true restartCount: 6 started: true state: running: startedAt: "2025-06-19T10:41:37Z" volumeMounts: - mountPath: /postgresql/volume_data name: data - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled - containerID: containerd://fe46f97f33678fa6ae8cf97d6d4e3035f3a031c1935e8e3a6e427d99071bf81e image: docker.io/apecloud/postgres:14.7 imageID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c lastState: terminated: containerID: containerd://820956b0b3afbf9bba2d6074479311619ab1eef40d8bc102418383e8e014485f exitCode: 2 finishedAt: "2025-06-19T10:41:15Z" reason: Error startedAt: "2025-06-19T10:41:15Z" name: postgresql ready: true restartCount: 7 started: true state: running: startedAt: "2025-06-19T10:41:37Z" volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /postgresql/volume_data name: data - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /kb-scripts name: scripts - mountPath: /kubeblocks name: kubeblocks - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled hostIP: 172.31.6.29 hostIPs: - ip: 172.31.6.29 initContainerStatuses: - containerID: containerd://5e5278ee7a8be5bdf9eef18a2c7b8cc17cdece0282eb90c489143095bc5c5e9f image: docker.io/apecloud/syncer:0.3.6 imageID: docker.io/apecloud/syncer@sha256:d8cb38ff8f9e025e6211725bb61ac1cbaadb10dc5e9dc6a2613ef09f7a42f26e lastState: *** name: init-syncer ready: true restartCount: 0 started: false state: terminated: containerID: containerd://5e5278ee7a8be5bdf9eef18a2c7b8cc17cdece0282eb90c489143095bc5c5e9f exitCode: 0 finishedAt: "2025-06-19T10:31:44Z" reason: Completed startedAt: "2025-06-19T10:31:44Z" volumeMounts: - mountPath: /kubeblocks name: kubeblocks - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled - containerID: containerd://4d462f4a83b1519c07e61de32cee6a50957374b57f9975eceeec7723ab1fa807 image: docker.io/apecloud/kubeblocks-tools:0.9.4 imageID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 lastState: *** name: install-config-manager-tool ready: true restartCount: 0 started: false state: terminated: containerID: containerd://4d462f4a83b1519c07e61de32cee6a50957374b57f9975eceeec7723ab1fa807 exitCode: 0 finishedAt: "2025-06-19T10:31:45Z" reason: Completed startedAt: "2025-06-19T10:31:45Z" volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled phase: Running podIP: 172.31.11.240 podIPs: - ip: 172.31.11.240 qosClass: Burstable startTime: "2025-06-19T10:31:40Z" ------------------------------------------------------------------------------------------------------------------ `kubectl get pod vanilla-kzgbtp-postgresql-1 -o yaml --namespace ns-auxmm ` apiVersion: v1 kind: Pod metadata: annotations: apps.kubeblocks.io/component-replicas: "2" kubeblocks.io/restart: "2025-06-19T11:16:07Z" creationTimestamp: "2025-06-19T11:16:08Z" labels: app.kubernetes.io/component: vanilla-postgresql-14 app.kubernetes.io/instance: vanilla-kzgbtp app.kubernetes.io/managed-by: kubeblocks app.kubernetes.io/name: vanilla-postgresql-14 app.kubernetes.io/version: vanilla-postgresql-14 apps.kubeblocks.io/cluster-uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 apps.kubeblocks.io/component-name: postgresql apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-1 clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" componentdefinition.kubeblocks.io/name: vanilla-postgresql-14 controller-revision-hash: 6bfcc759f workloads.kubeblocks.io/instance: vanilla-kzgbtp-postgresql workloads.kubeblocks.io/managed-by: InstanceSet name: vanilla-kzgbtp-postgresql-1 namespace: ns-auxmm ownerReferences: - apiVersion: workloads.kubeblocks.io/v1alpha1 blockOwnerDeletion: true controller: true kind: InstanceSet name: vanilla-kzgbtp-postgresql uid: 7be55aad-9aa7-48fb-b705-0b349c86f2aa resourceVersion: "81679" uid: f6124d06-79c8-478c-9124-c3bb4a090337 spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: kb-data operator: In values: - "true" weight: 100 containers: - command: - /kubeblocks/syncer - --port - "3601" - -- - docker-entrypoint.sh - --config-file=/postgresql/volume_data/conf/postgresql.conf - --hba_file=/postgresql/volume_data/conf/pg_hba.conf env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: ALLOW_NOSSL value: "true" - name: POSTGRESQL_PORT_NUMBER value: "5432" - name: PGDATA value: /postgresql/volume_data/data - name: PGCONF value: /postgresql/volume_data/conf - name: POSTGRESQL_MOUNTED_CONF_DIR value: /postgresql/mount_conf - name: PGUSER value: $(POSTGRES_USER) - name: PGPASSWORD value: $(POSTGRES_PASSWORD) - name: KB_ENGINE_TYPE value: vanilla-postgresql envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/postgres:14.7 imagePullPolicy: IfNotPresent name: postgresql ports: - containerPort: 5432 name: tcp-postgresql protocol: TCP resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m securityContext: runAsUser: 0 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /postgresql/volume_data name: data - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /kb-scripts name: scripts - mountPath: /kubeblocks name: kubeblocks - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true - command: - lorry - --port - "3501" - --grpcport - "50001" env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: KB_RSM_ROLE_PROBE_PERIOD value: "1" - name: KB_BUILTIN_HANDLER value: postgresql - name: KB_SERVICE_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_SERVICE_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_SERVICE_PORT value: "5432" - name: KB_DATA_PATH value: /postgresql/volume_data - name: KB_RSM_ACTION_SVC_LIST value: "null" - name: KB_RSM_ROLE_UPDATE_MECHANISM value: DirectAPIServerEventUpdate - name: KB_RSM_ROLE_PROBE_TIMEOUT value: "1" - name: KB_CLUSTER_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.labels['app.kubernetes.io/instance'] - name: KB_COMP_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.labels['apps.kubeblocks.io/component-name'] - name: KB_SERVICE_CHARACTER_TYPE value: postgresql envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/kubeblocks-tools:0.9.4 imagePullPolicy: IfNotPresent name: lorry ports: - containerPort: 3501 name: lorry-http-port protocol: TCP - containerPort: 50001 name: lorry-grpc-port protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /v1.0/checkrole port: 3501 scheme: HTTP periodSeconds: 1 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" startupProbe: failureThreshold: 3 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 3501 timeoutSeconds: 1 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/volume_data name: data - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true - args: - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) - /kb_tools/reloader - --log-level - info - --operator-update-enable - --tcp - "9901" - --config - /opt/config-manager/config-manager.yaml command: - env env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: CONFIG_MANAGER_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: TOOLS_PATH value: /opt/kb-tools/reload/vanilla-postgresql-configuration:/opt/config-manager:/kb_tools envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/postgres:14.7 imagePullPolicy: IfNotPresent name: config-manager ports: - containerPort: 9901 name: config-manager protocol: TCP resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostname: vanilla-kzgbtp-postgresql-1 initContainers: - command: - sh - -c - cp -r /bin/syncer /kubeblocks/ env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false image: docker.io/apecloud/syncer:0.3.6 imagePullPolicy: IfNotPresent name: init-syncer resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /kubeblocks name: kubeblocks - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true - command: - cp - /bin/reloader - /kb_tools env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false image: docker.io/apecloud/kubeblocks-tools:0.9.4 imagePullPolicy: IfNotPresent name: install-config-manager-tool resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true nodeName: ip-172-31-14-226.us-west-2.compute.internal preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 103 runAsGroup: 103 runAsUser: 0 serviceAccount: kb-vanilla-kzgbtp serviceAccountName: kb-vanilla-kzgbtp subdomain: vanilla-kzgbtp-postgresql-headless terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule key: kb-data operator: Equal value: "true" - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - emptyDir: medium: Memory sizeLimit: 644245094400m name: dshm - configMap: defaultMode: 292 name: vanilla-kzgbtp-postgresql-vanilla-postgresql-configuration name: postgresql-config - configMap: defaultMode: 365 name: vanilla-kzgbtp-postgresql-vanilla-postgresql-scripts name: scripts - configMap: defaultMode: 493 name: sidecar-vanilla-postgresql-reload-tools-script-vanilla-kzgbtp name: cm-script-vanilla-postgresql-configuration - configMap: defaultMode: 493 name: sidecar-vanilla-kzgbtp-postgresql-config-manager-config name: config-manager-config - emptyDir: *** name: kb-tools - name: data persistentVolumeClaim: claimName: data-vanilla-kzgbtp-postgresql-1 - emptyDir: *** name: kubeblocks - name: kube-api-access-85ck4 projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2025-06-19T11:16:18Z" status: "True" type: PodReadyToStartContainers - lastProbeTime: null lastTransitionTime: "2025-06-19T11:16:19Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2025-06-19T11:32:25Z" message: 'containers with unready status: [postgresql]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2025-06-19T11:32:25Z" message: 'containers with unready status: [postgresql]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2025-06-19T11:16:08Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://7d0368d1912443bfccfab273c4db8c6a06a035a7519a254098b5292289b6308f image: docker.io/apecloud/postgres:14.7 imageID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c lastState: *** name: config-manager ready: true restartCount: 0 started: true state: running: startedAt: "2025-06-19T11:16:19Z" volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true recursiveReadOnly: Disabled - containerID: containerd://47ba077bb795b802507fb7e425c7ae928a6f4616d32538bcd407806a9db33d42 image: docker.io/apecloud/kubeblocks-tools:0.9.4 imageID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 lastState: *** name: lorry ready: true restartCount: 0 started: true state: running: startedAt: "2025-06-19T11:16:19Z" volumeMounts: - mountPath: /postgresql/volume_data name: data - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true recursiveReadOnly: Disabled - containerID: containerd://c95f2793316381e780eceab19830ba0d5cc0adbb1fd766a570cd988b4e9e2eb3 image: docker.io/apecloud/postgres:14.7 imageID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c lastState: terminated: containerID: containerd://c95f2793316381e780eceab19830ba0d5cc0adbb1fd766a570cd988b4e9e2eb3 exitCode: 2 finishedAt: "2025-06-19T11:32:25Z" reason: Error startedAt: "2025-06-19T11:32:24Z" name: postgresql ready: false restartCount: 8 started: false state: waiting: message: back-off 5m0s restarting failed container=postgresql pod=vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) reason: CrashLoopBackOff volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /postgresql/volume_data name: data - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /kb-scripts name: scripts - mountPath: /kubeblocks name: kubeblocks - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true recursiveReadOnly: Disabled hostIP: 172.31.14.226 hostIPs: - ip: 172.31.14.226 initContainerStatuses: - containerID: containerd://de21f4e9dcc84ba2738b4656cef895418bef7448e41a194d8c77a0f3fc55ad99 image: docker.io/apecloud/syncer:0.3.6 imageID: docker.io/apecloud/syncer@sha256:d8cb38ff8f9e025e6211725bb61ac1cbaadb10dc5e9dc6a2613ef09f7a42f26e lastState: *** name: init-syncer ready: true restartCount: 0 started: false state: terminated: containerID: containerd://de21f4e9dcc84ba2738b4656cef895418bef7448e41a194d8c77a0f3fc55ad99 exitCode: 0 finishedAt: "2025-06-19T11:16:18Z" reason: Completed startedAt: "2025-06-19T11:16:18Z" volumeMounts: - mountPath: /kubeblocks name: kubeblocks - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true recursiveReadOnly: Disabled - containerID: containerd://ab9eb3a43c6faf6a74523296dd9f72dd0f8ee346b740b90f591a010233b8dcbb image: docker.io/apecloud/kubeblocks-tools:0.9.4 imageID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 lastState: *** name: install-config-manager-tool ready: true restartCount: 0 started: false state: terminated: containerID: containerd://ab9eb3a43c6faf6a74523296dd9f72dd0f8ee346b740b90f591a010233b8dcbb exitCode: 0 finishedAt: "2025-06-19T11:16:18Z" reason: Completed startedAt: "2025-06-19T11:16:18Z" volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-85ck4 readOnly: true recursiveReadOnly: Disabled phase: Running podIP: 172.31.6.187 podIPs: - ip: 172.31.6.187 qosClass: Burstable startTime: "2025-06-19T11:16:08Z" ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe pod vanilla-kzgbtp-postgresql-0 vanilla-kzgbtp-postgresql-1-------------------------------------- `kubectl describe pod vanilla-kzgbtp-postgresql-0 --namespace ns-auxmm ` Name: vanilla-kzgbtp-postgresql-0 Namespace: ns-auxmm Priority: 0 Service Account: kb-vanilla-kzgbtp Node: ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Start Time: Thu, 19 Jun 2025 18:31:40 +0800 Labels: app.kubernetes.io/component=vanilla-postgresql-14 app.kubernetes.io/instance=vanilla-kzgbtp app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=vanilla-postgresql-14 app.kubernetes.io/version=vanilla-postgresql-14 apps.kubeblocks.io/cluster-uid=9fe00cca-f01b-4f4e-8e1c-f33f8591d154 apps.kubeblocks.io/component-name=postgresql apps.kubeblocks.io/pod-name=vanilla-kzgbtp-postgresql-0 clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= componentdefinition.kubeblocks.io/name=vanilla-postgresql-14 controller-revision-hash=6c8d9d7bf5 kubeblocks.io/role=primary workloads.kubeblocks.io/access-mode=ReadWrite workloads.kubeblocks.io/instance=vanilla-kzgbtp-postgresql workloads.kubeblocks.io/managed-by=InstanceSet Annotations: apps.kubeblocks.io/component-replicas: 2 apps.kubeblocks.io/last-role-snapshot-version: 1750329814096971 Status: Running IP: 172.31.11.240 IPs: IP: 172.31.11.240 Controlled By: InstanceSet/vanilla-kzgbtp-postgresql Init Containers: init-syncer: Container ID: containerd://5e5278ee7a8be5bdf9eef18a2c7b8cc17cdece0282eb90c489143095bc5c5e9f Image: docker.io/apecloud/syncer:0.3.6 Image ID: docker.io/apecloud/syncer@sha256:d8cb38ff8f9e025e6211725bb61ac1cbaadb10dc5e9dc6a2613ef09f7a42f26e Port: Host Port: Command: sh -c cp -r /bin/syncer /kubeblocks/ State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 19 Jun 2025 18:31:44 +0800 Finished: Thu, 19 Jun 2025 18:31:44 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc Mounts: /kubeblocks from kubeblocks (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) install-config-manager-tool: Container ID: containerd://4d462f4a83b1519c07e61de32cee6a50957374b57f9975eceeec7723ab1fa807 Image: docker.io/apecloud/kubeblocks-tools:0.9.4 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 Port: Host Port: Command: cp /bin/reloader /kb_tools State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 19 Jun 2025 18:31:45 +0800 Finished: Thu, 19 Jun 2025 18:31:45 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc Mounts: /kb_tools from kb-tools (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/vanilla-postgresql-configuration from cm-script-vanilla-postgresql-configuration (rw) /postgresql/mount_conf from postgresql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) Containers: postgresql: Container ID: containerd://fe46f97f33678fa6ae8cf97d6d4e3035f3a031c1935e8e3a6e427d99071bf81e Image: docker.io/apecloud/postgres:14.7 Image ID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c Port: 5432/TCP Host Port: 0/TCP Command: /kubeblocks/syncer --port 3601 -- docker-entrypoint.sh --config-file=/postgresql/volume_data/conf/postgresql.conf --hba_file=/postgresql/volume_data/conf/pg_hba.conf State: Running Started: Thu, 19 Jun 2025 18:41:37 +0800 Last State: Terminated Reason: Error Exit Code: 2 Started: Thu, 19 Jun 2025 18:41:15 +0800 Finished: Thu, 19 Jun 2025 18:41:15 +0800 Ready: True Restart Count: 7 Limits: cpu: 600m memory: 644245094400m Requests: cpu: 600m memory: 644245094400m Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc ALLOW_NOSSL: true POSTGRESQL_PORT_NUMBER: 5432 PGDATA: /postgresql/volume_data/data PGCONF: /postgresql/volume_data/conf POSTGRESQL_MOUNTED_CONF_DIR: /postgresql/mount_conf PGUSER: $(POSTGRES_USER) PGPASSWORD: $(POSTGRES_PASSWORD) KB_ENGINE_TYPE: vanilla-postgresql Mounts: /dev/shm from dshm (rw) /kb-scripts from scripts (rw) /kb_tools from kb-tools (rw) /kubeblocks from kubeblocks (rw) /postgresql/mount_conf from postgresql-config (rw) /postgresql/volume_data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) lorry: Container ID: containerd://fa38d6cf08bb686721e3f36bc4e802c5b83c45ea15d8c4b62134e4f1153a1a6d Image: docker.io/apecloud/kubeblocks-tools:0.9.4 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: lorry --port 3501 --grpcport 50001 State: Running Started: Thu, 19 Jun 2025 18:41:37 +0800 Last State: Terminated Reason: StartError Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "lorry": executable file not found in $PATH: unknown Exit Code: 128 Started: Thu, 01 Jan 1970 08:00:00 +0800 Finished: Thu, 19 Jun 2025 18:41:15 +0800 Ready: True Restart Count: 6 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Readiness: http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc KB_RSM_ROLE_PROBE_PERIOD: 1 KB_BUILTIN_HANDLER: postgresql KB_SERVICE_USER: Optional: false KB_SERVICE_PASSWORD: Optional: false KB_SERVICE_PORT: 5432 KB_DATA_PATH: /postgresql/volume_data KB_RSM_ACTION_SVC_LIST: null KB_RSM_ROLE_UPDATE_MECHANISM: DirectAPIServerEventUpdate KB_RSM_ROLE_PROBE_TIMEOUT: 1 KB_CLUSTER_NAME: (v1:metadata.labels['app.kubernetes.io/instance']) KB_COMP_NAME: (v1:metadata.labels['apps.kubeblocks.io/component-name']) KB_SERVICE_CHARACTER_TYPE: postgresql Mounts: /postgresql/volume_data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) config-manager: Container ID: containerd://4a559d5430038eadb6ec4f2b6654901ec74628489218239605b3f1d1ae551206 Image: docker.io/apecloud/postgres:14.7 Image ID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c Port: 9901/TCP Host Port: 0/TCP Command: env Args: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) /kb_tools/reloader --log-level info --operator-update-enable --tcp 9901 --config /opt/config-manager/config-manager.yaml State: Running Started: Thu, 19 Jun 2025 18:41:37 +0800 Last State: Terminated Reason: StartError Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "env": executable file not found in $PATH: unknown Exit Code: 128 Started: Thu, 01 Jan 1970 08:00:00 +0800 Finished: Thu, 19 Jun 2025 18:41:16 +0800 Ready: True Restart Count: 6 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc CONFIG_MANAGER_POD_IP: (v1:status.podIP) TOOLS_PATH: /opt/kb-tools/reload/vanilla-postgresql-configuration:/opt/config-manager:/kb_tools Mounts: /kb_tools from kb-tools (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/vanilla-postgresql-configuration from cm-script-vanilla-postgresql-configuration (rw) /postgresql/mount_conf from postgresql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: dshm: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: 644245094400m postgresql-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: vanilla-kzgbtp-postgresql-vanilla-postgresql-configuration Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: vanilla-kzgbtp-postgresql-vanilla-postgresql-scripts Optional: false cm-script-vanilla-postgresql-configuration: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-vanilla-postgresql-reload-tools-script-vanilla-kzgbtp Optional: false config-manager-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-vanilla-kzgbtp-postgresql-config-manager-config Optional: false kb-tools: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-vanilla-kzgbtp-postgresql-0 ReadOnly: false kubeblocks: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-lfdrn: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kb-data=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal checkRole 59m lorry ***"operation":"checkRole","originalRole":"primary"*** Normal checkRole 59m lorry ***"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"*** Normal healthyCheck 59m syncer ***"member":"vanilla-kzgbtp-postgresql-0","message":"Failover happens, i am the healthiest member, attempt to take the leader","operation":"healthyCheck"*** Normal checkRole 59m lorry ***"event":"Success","operation":"checkRole","originalRole":"secondary","role":"***\"term\":\"1750329288340760\",\"PodRoleNamePairs\":[***\"podName\":\"vanilla-kzgbtp-postgresql-0\",\"roleName\":\"primary\",\"podUid\":\"f3b3a75d-ecb7-4978-baeb-a7597f9d8a14\"***]***"*** Normal checkRole 59m lorry ***"operation":"checkRole","originalRole":"primary"*** Normal Created 59m (x3 over 62m) kubelet Created container: postgresql Normal Started 59m (x3 over 62m) kubelet Started container postgresql Normal Pulled 59m (x3 over 62m) kubelet Container image "docker.io/apecloud/postgres:14.7" already present on machine Normal checkRole 59m lorry ***"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"*** Normal healthyCheck 56m syncer ***"member":"vanilla-kzgbtp-postgresql-0","message":"Failover happens, i am the healthiest member, attempt to take the leader","operation":"healthyCheck"*** Normal checkRole 56m lorry ***"event":"Success","operation":"checkRole","originalRole":"secondary","role":"***\"term\":\"1750329503424873\",\"PodRoleNamePairs\":[***\"podName\":\"vanilla-kzgbtp-postgresql-0\",\"roleName\":\"primary\",\"podUid\":\"f3b3a75d-ecb7-4978-baeb-a7597f9d8a14\"***]***"*** Normal Killing 55m kubelet Container lorry definition changed, will be restarted Normal Pulling 55m kubelet Pulling image "gcr.io/google-containers/pause:latest" Normal Killing 55m kubelet Container config-manager definition changed, will be restarted Normal Killing 55m kubelet Container postgresql definition changed, will be restarted Normal Pulled 54m kubelet Successfully pulled image "gcr.io/google-containers/pause:latest" in 1.37s (1.37s including waiting). Image size: 72335 bytes. Warning Failed 54m (x2 over 54m) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "env": executable file not found in $PATH: unknown Warning Unhealthy 54m (x3 over 55m) kubelet Readiness probe failed: Get "http://172.31.11.240:3501/v1.0/checkrole": dial tcp 172.31.11.240:3501: connect: connection refused Warning BackOff 54m (x5 over 54m) kubelet Back-off restarting failed container lorry in pod vanilla-kzgbtp-postgresql-0_ns-auxmm(f3b3a75d-ecb7-4978-baeb-a7597f9d8a14) Warning BackOff 54m (x5 over 54m) kubelet Back-off restarting failed container config-manager in pod vanilla-kzgbtp-postgresql-0_ns-auxmm(f3b3a75d-ecb7-4978-baeb-a7597f9d8a14) Warning BackOff 54m (x7 over 59m) kubelet Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-0_ns-auxmm(f3b3a75d-ecb7-4978-baeb-a7597f9d8a14) Warning Failed 54m (x4 over 54m) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "lorry": executable file not found in $PATH: unknown Normal checkRole 52m lorry ***"event":"Success","operation":"checkRole","originalRole":"waitForStart","role":"secondary"*** Normal healthyCheck 51m syncer ***"member":"vanilla-kzgbtp-postgresql-0","message":"Failover happens, i am the healthiest member, attempt to take the leader","operation":"healthyCheck"*** Normal checkRole 51m lorry ***"event":"Success","operation":"checkRole","originalRole":"secondary","role":"***\"term\":\"1750329814096971\",\"PodRoleNamePairs\":[***\"podName\":\"vanilla-kzgbtp-postgresql-0\",\"roleName\":\"primary\",\"podUid\":\"f3b3a75d-ecb7-4978-baeb-a7597f9d8a14\"***,***\"podName\":\"vanilla-kzgbtp-postgresql-1\",\"podUid\":\"5ddfe019-0f5b-441f-8519-bcac8f4bf393\"***]***"*** Normal checkRole 51m lorry ***"event":"Success","operation":"checkRole","originalRole":"secondary","role":"***\"term\":\"1750329813058427\",\"PodRoleNamePairs\":[***\"podName\":\"vanilla-kzgbtp-postgresql-0\",\"roleName\":\"primary\",\"podUid\":\"f3b3a75d-ecb7-4978-baeb-a7597f9d8a14\"***,***\"podName\":\"vanilla-kzgbtp-postgresql-1\",\"podUid\":\"5ddfe019-0f5b-441f-8519-bcac8f4bf393\"***]***"*** Normal FileSystemResizeSuccessful 32m kubelet MountVolume.NodeExpandVolume succeeded for volume "pvc-14a027e8-e9ed-493e-95fe-36324e4f3d1f" ip-172-31-6-29.us-west-2.compute.internal Warning Unhealthy 19m (x1206 over 51m) kubelet Readiness probe failed: Get "http://172.31.11.240:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) ------------------------------------------------------------------------------------------------------------------ `kubectl describe pod vanilla-kzgbtp-postgresql-1 --namespace ns-auxmm ` Name: vanilla-kzgbtp-postgresql-1 Namespace: ns-auxmm Priority: 0 Service Account: kb-vanilla-kzgbtp Node: ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Start Time: Thu, 19 Jun 2025 19:16:08 +0800 Labels: app.kubernetes.io/component=vanilla-postgresql-14 app.kubernetes.io/instance=vanilla-kzgbtp app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=vanilla-postgresql-14 app.kubernetes.io/version=vanilla-postgresql-14 apps.kubeblocks.io/cluster-uid=9fe00cca-f01b-4f4e-8e1c-f33f8591d154 apps.kubeblocks.io/component-name=postgresql apps.kubeblocks.io/pod-name=vanilla-kzgbtp-postgresql-1 clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= componentdefinition.kubeblocks.io/name=vanilla-postgresql-14 controller-revision-hash=6bfcc759f workloads.kubeblocks.io/instance=vanilla-kzgbtp-postgresql workloads.kubeblocks.io/managed-by=InstanceSet Annotations: apps.kubeblocks.io/component-replicas: 2 kubeblocks.io/restart: 2025-06-19T11:16:07Z Status: Running IP: 172.31.6.187 IPs: IP: 172.31.6.187 Controlled By: InstanceSet/vanilla-kzgbtp-postgresql Init Containers: init-syncer: Container ID: containerd://de21f4e9dcc84ba2738b4656cef895418bef7448e41a194d8c77a0f3fc55ad99 Image: docker.io/apecloud/syncer:0.3.6 Image ID: docker.io/apecloud/syncer@sha256:d8cb38ff8f9e025e6211725bb61ac1cbaadb10dc5e9dc6a2613ef09f7a42f26e Port: Host Port: Command: sh -c cp -r /bin/syncer /kubeblocks/ State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 19 Jun 2025 19:16:18 +0800 Finished: Thu, 19 Jun 2025 19:16:18 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc Mounts: /kubeblocks from kubeblocks (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-85ck4 (ro) install-config-manager-tool: Container ID: containerd://ab9eb3a43c6faf6a74523296dd9f72dd0f8ee346b740b90f591a010233b8dcbb Image: docker.io/apecloud/kubeblocks-tools:0.9.4 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 Port: Host Port: Command: cp /bin/reloader /kb_tools State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 19 Jun 2025 19:16:18 +0800 Finished: Thu, 19 Jun 2025 19:16:18 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc Mounts: /kb_tools from kb-tools (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/vanilla-postgresql-configuration from cm-script-vanilla-postgresql-configuration (rw) /postgresql/mount_conf from postgresql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-85ck4 (ro) Containers: postgresql: Container ID: containerd://c95f2793316381e780eceab19830ba0d5cc0adbb1fd766a570cd988b4e9e2eb3 Image: docker.io/apecloud/postgres:14.7 Image ID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c Port: 5432/TCP Host Port: 0/TCP Command: /kubeblocks/syncer --port 3601 -- docker-entrypoint.sh --config-file=/postgresql/volume_data/conf/postgresql.conf --hba_file=/postgresql/volume_data/conf/pg_hba.conf State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Thu, 19 Jun 2025 19:32:24 +0800 Finished: Thu, 19 Jun 2025 19:32:25 +0800 Ready: False Restart Count: 8 Limits: cpu: 600m memory: 644245094400m Requests: cpu: 600m memory: 644245094400m Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc ALLOW_NOSSL: true POSTGRESQL_PORT_NUMBER: 5432 PGDATA: /postgresql/volume_data/data PGCONF: /postgresql/volume_data/conf POSTGRESQL_MOUNTED_CONF_DIR: /postgresql/mount_conf PGUSER: $(POSTGRES_USER) PGPASSWORD: $(POSTGRES_PASSWORD) KB_ENGINE_TYPE: vanilla-postgresql Mounts: /dev/shm from dshm (rw) /kb-scripts from scripts (rw) /kb_tools from kb-tools (rw) /kubeblocks from kubeblocks (rw) /postgresql/mount_conf from postgresql-config (rw) /postgresql/volume_data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-85ck4 (ro) lorry: Container ID: containerd://47ba077bb795b802507fb7e425c7ae928a6f4616d32538bcd407806a9db33d42 Image: docker.io/apecloud/kubeblocks-tools:0.9.4 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: lorry --port 3501 --grpcport 50001 State: Running Started: Thu, 19 Jun 2025 19:16:19 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Readiness: http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc KB_RSM_ROLE_PROBE_PERIOD: 1 KB_BUILTIN_HANDLER: postgresql KB_SERVICE_USER: Optional: false KB_SERVICE_PASSWORD: Optional: false KB_SERVICE_PORT: 5432 KB_DATA_PATH: /postgresql/volume_data KB_RSM_ACTION_SVC_LIST: null KB_RSM_ROLE_UPDATE_MECHANISM: DirectAPIServerEventUpdate KB_RSM_ROLE_PROBE_TIMEOUT: 1 KB_CLUSTER_NAME: (v1:metadata.labels['app.kubernetes.io/instance']) KB_COMP_NAME: (v1:metadata.labels['apps.kubeblocks.io/component-name']) KB_SERVICE_CHARACTER_TYPE: postgresql Mounts: /postgresql/volume_data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-85ck4 (ro) config-manager: Container ID: containerd://7d0368d1912443bfccfab273c4db8c6a06a035a7519a254098b5292289b6308f Image: docker.io/apecloud/postgres:14.7 Image ID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c Port: 9901/TCP Host Port: 0/TCP Command: env Args: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) /kb_tools/reloader --log-level info --operator-update-enable --tcp 9901 --config /opt/config-manager/config-manager.yaml State: Running Started: Thu, 19 Jun 2025 19:16:19 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc CONFIG_MANAGER_POD_IP: (v1:status.podIP) TOOLS_PATH: /opt/kb-tools/reload/vanilla-postgresql-configuration:/opt/config-manager:/kb_tools Mounts: /kb_tools from kb-tools (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/vanilla-postgresql-configuration from cm-script-vanilla-postgresql-configuration (rw) /postgresql/mount_conf from postgresql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-85ck4 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: dshm: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: 644245094400m postgresql-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: vanilla-kzgbtp-postgresql-vanilla-postgresql-configuration Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: vanilla-kzgbtp-postgresql-vanilla-postgresql-scripts Optional: false cm-script-vanilla-postgresql-configuration: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-vanilla-postgresql-reload-tools-script-vanilla-kzgbtp Optional: false config-manager-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-vanilla-kzgbtp-postgresql-config-manager-config Optional: false kb-tools: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-vanilla-kzgbtp-postgresql-1 ReadOnly: false kubeblocks: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-85ck4: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kb-data=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 18m default-scheduler Successfully assigned ns-auxmm/vanilla-kzgbtp-postgresql-1 to ip-172-31-14-226.us-west-2.compute.internal Normal Pulled 18m kubelet Container image "docker.io/apecloud/syncer:0.3.6" already present on machine Normal Created 18m kubelet Created container: init-syncer Normal Started 18m kubelet Started container init-syncer Normal Pulled 18m kubelet Container image "docker.io/apecloud/kubeblocks-tools:0.9.4" already present on machine Normal Created 18m kubelet Created container: install-config-manager-tool Normal Started 18m kubelet Started container install-config-manager-tool Normal Started 18m kubelet Started container lorry Normal Pulled 18m kubelet Container image "docker.io/apecloud/kubeblocks-tools:0.9.4" already present on machine Normal Created 18m kubelet Created container: lorry Normal Pulled 18m kubelet Container image "docker.io/apecloud/postgres:14.7" already present on machine Normal Created 18m kubelet Created container: config-manager Normal Started 18m kubelet Started container config-manager Normal Created 17m (x4 over 18m) kubelet Created container: postgresql Normal Started 17m (x4 over 18m) kubelet Started container postgresql Warning BackOff 3m8s (x75 over 18m) kubelet Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) Normal Pulled 2m16s (x9 over 18m) kubelet Container image "docker.io/apecloud/postgres:14.7" already present on machine ------------------------------------------------------------------------------------------------------------------ --------------------------------------pod vanilla-kzgbtp-postgresql-0 vanilla-kzgbtp-postgresql-1-------------------------------------- `kubectl logs vanilla-kzgbtp-postgresql-0 --namespace ns-auxmm --tail 500` Defaulted container "postgresql" out of: postgresql, lorry, config-manager, init-syncer (init), install-config-manager-tool (init) == DB ERR == 2025-06-19 11:32:38.185 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:39.186 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:40.185 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:41.186 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:42.187 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:43.220 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == 2025-06-19T11:32:43Z INFO HA This member is Cluster's leader 2025-06-19T11:32:43Z DEBUG HA Refresh leader ttl == DB ERR == 2025-06-19 11:32:44.186 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:45.186 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:46.187 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:47.188 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:48.187 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:49.187 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:50.188 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:51.189 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:52.189 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:53.189 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:54.255 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:55.190 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:56.190 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:57.190 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:58.190 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:32:59.190 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:00.190 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:01.191 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:02.192 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:03.192 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:04.192 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:05.216 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:06.194 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:07.195 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:08.195 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:09.196 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:10.196 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:11.196 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:12.197 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:13.198 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:14.198 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:15.197 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:16.224 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:17.199 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:18.200 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:19.200 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:20.199 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:21.200 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:22.201 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:23.202 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:24.204 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:25.204 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:26.203 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:27.287 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:28.204 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:29.205 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:30.205 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:31.206 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:32.207 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:33.207 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:34.208 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:35.209 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:36.209 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:37.209 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:38.238 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:39.210 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:40.210 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:41.210 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:42.212 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:43.213 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == 2025-06-19T11:33:43Z INFO HA This member is Cluster's leader 2025-06-19T11:33:43Z DEBUG HA Refresh leader ttl == DB ERR == 2025-06-19 11:33:44.212 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:45.212 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:46.214 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:47.215 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:48.216 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:49.242 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:50.216 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:51.216 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:52.218 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:53.218 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:54.218 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:55.218 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:56.219 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:57.219 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:58.221 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:33:59.221 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:00.248 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:01.221 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:02.222 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:03.223 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:04.224 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:05.225 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:06.224 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:07.225 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:08.226 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:09.226 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:10.226 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:11.254 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:12.227 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:13.227 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:14.228 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:15.228 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:16.229 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:17.230 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:18.230 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:19.231 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:20.231 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:21.232 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:22.255 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:23.232 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:24.234 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:25.235 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:26.237 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:27.236 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:28.237 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:29.239 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:30.238 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:31.250 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:32.238 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:33.269 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:34.242 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:35.242 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:36.243 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:37.243 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:38.244 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:39.244 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:40.244 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 11:34:41.244 GMT [47] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == ------------------------------------------------------------------------------------------------------------------ `kubectl logs vanilla-kzgbtp-postgresql-1 --namespace ns-auxmm --tail 500` Defaulted container "postgresql" out of: postgresql, lorry, config-manager, init-syncer (init), install-config-manager-tool (init) 2025-06-19T11:32:24Z DEBUG Starting syncer ***"version": "4e24a0c83c87ff35927d5730f6616bc6f00e2f88"*** 2025-06-19T11:32:24Z INFO Initialize DB manager 2025-06-19T11:32:24Z DEBUG KB_WORKLOAD_TYPE ENV not set 2025-06-19T11:32:24Z INFO HTTPServer Starting HTTP Server 2025-06-19T11:32:24Z INFO HTTPServer API route path ***"method": "POST", "path": ["/v1.0/rebuild", "/v1.0/start", "/v1.0/stop", "/v1.0/switchover"]*** 2025-06-19T11:32:24Z INFO HTTPServer API route path ***"method": "GET", "path": ["/v1.0/datasync", "/v1.0/getrole"]*** 2025-06-19T11:32:24Z INFO HTTPServer http server ***"listen address": "0.0.0.0", "port": 3601*** 2025-06-19T11:32:24Z INFO HA HA starting 2025-06-19T11:32:24Z INFO pinger Waiting for dns resolution to be ready 2025-06-19T11:32:24Z INFO pinger dns resolution is ready ***"dns": "vanilla-kzgbtp-postgresql-1.vanilla-kzgbtp-postgresql-headless.ns-auxmm.svc"*** 2025-06-19T11:32:24Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=vanilla-kzgbtp,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=postgresql 2025-06-19T11:32:24Z INFO DCS-K8S podlist: 2 2025-06-19T11:32:24Z DEBUG HA cluster info ***"cluster": ***"ClusterCompName":"vanilla-kzgbtp-postgresql","Namespace":"ns-auxmm","Replicas":2,"HaConfig":***"DeleteMembers":***,"Leader":***"DBState":***"OpTimestamp":168091864,"Extra":***"replication_mode":"off","timeline":"10"***,"Index":"81526","Name":"vanilla-kzgbtp-postgresql-0","AcquireTime":1750331751,"RenewTime":1750332744,"TTL":15,"Resource":***"metadata":***"name":"vanilla-kzgbtp-postgresql-leader","namespace":"ns-auxmm","uid":"f9f4e470-b0b3-439f-bbee-fa6df200607b","resourceVersion":"81526","creationTimestamp":"2025-06-19T10:16:02Z","labels":***"app.kubernetes.io/instance":"vanilla-kzgbtp","app.kubernetes.io/managed-by":"kubeblocks","apps.kubeblocks.io/component-name":"postgresql"***,"annotations":***"acquire-time":"1750331751","dbstate":"***\"OpTimestamp\":168091864,\"Extra\":***\"replication_mode\":\"off\",\"timeline\":\"10\"***","leader":"vanilla-kzgbtp-postgresql-0","renew-time":"1750332744","ttl":"15"***,"ownerReferences":[***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","name":"vanilla-kzgbtp","uid":"9fe00cca-f01b-4f4e-8e1c-f33f8591d154"***],"managedFields":[***"manager":"syncer","operation":"Update","apiVersion":"v1","time":"2025-06-19T11:32:24Z","fieldsType":"FieldsV1","fieldsV1":***"f:metadata":***"f:annotations":***".":***,"f:acquire-time":***,"f:dbstate":***,"f:leader":***,"f:renew-time":***,"f:ttl":***,"f:labels":***".":***,"f:app.kubernetes.io/instance":***,"f:app.kubernetes.io/managed-by":***,"f:apps.kubeblocks.io/component-name":***,"f:ownerReferences":***".":***,"k:***\"uid\":\"9fe00cca-f01b-4f4e-8e1c-f33f8591d154\"***":***]***,"Members":[***"Index":"","Name":"vanilla-kzgbtp-postgresql-0","Role":"primary","PodIP":"172.31.11.240","DBPort":"5432","SyncerPort":"3601","UID":"f3b3a75d-ecb7-4978-baeb-a7597f9d8a14","ComponentName":"postgresql","UseIP":false***,***"Index":"","Name":"vanilla-kzgbtp-postgresql-1","Role":"","PodIP":"172.31.6.187","DBPort":"5432","SyncerPort":"3601","UID":"f6124d06-79c8-478c-9124-c3bb4a090337","ComponentName":"postgresql","UseIP":false***],"Switchover":null,"Extra":null,"Resource":***"metadata":***"name":"vanilla-kzgbtp","namespace":"ns-auxmm","uid":"9fe00cca-f01b-4f4e-8e1c-f33f8591d154","resourceVersion":"80945","generation":9,"creationTimestamp":"2025-06-19T10:15:26Z","labels":***"app.kubernetes.io/instance":"vanilla-kzgbtp","clusterdefinition.kubeblocks.io/name":"vanilla-postgresql","clusterversion.kubeblocks.io/name":""***,"annotations":***"kubeblocks.io/ops-request":"[***\"name\":\"vanilla-kzgbtp-restart-xpvgz\",\"type\":\"Restart\"***]","kubeblocks.io/reconcile":"2025-06-19T11:31:32.273461683Z","kubectl.kubernetes.io/last-applied-configuration":"***\"apiVersion\":\"apps.kubeblocks.io/v1alpha1\",\"kind\":\"Cluster\",\"metadata\":***\"annotations\":***,\"name\":\"vanilla-kzgbtp\",\"namespace\":\"ns-auxmm\"***,\"spec\":***\"clusterDefinitionRef\":\"vanilla-postgresql\",\"componentSpecs\":[***\"name\":\"postgresql\",\"replicas\":2,\"resources\":***\"limits\":***\"cpu\":\"500m\",\"memory\":\"0.5Gi\"***,\"requests\":***\"cpu\":\"500m\",\"memory\":\"0.5Gi\"***,\"serviceVersion\":\"14.7.0\",\"volumeClaimTemplates\":[***\"name\":\"data\",\"spec\":***\"accessModes\":[\"ReadWriteOnce\"],\"resources\":***\"requests\":***\"storage\":\"3Gi\"***,\"storageClassName\":null***]***],\"terminationPolicy\":\"Halt\",\"topology\":\"vanilla-postgresql\"***\n"***,"finalizers":["cluster.kubeblocks.io/finalizer"],"managedFields":[***"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2025-06-19T10:15:26Z","fieldsType":"FieldsV1","fieldsV1":***"f:metadata":***"f:annotations":***".":***,"f:kubectl.kubernetes.io/last-applied-configuration":***,"f:spec":***".":***,"f:clusterDefinitionRef":***,"f:terminationPolicy":***,"f:topology":***,***"manager":"kbcli","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2025-06-19T10:16:50Z","fieldsType":"FieldsV1","fieldsV1":***"f:metadata":***"f:labels":***"f:app.kubernetes.io/instance":***,***"manager":"kubeblocks","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2025-06-19T11:27:31Z","fieldsType":"FieldsV1","fieldsV1":***"f:status":***".":***,"f:clusterDefGeneration":***,"f:components":***".":***,"f:postgresql":***".":***,"f:message":***".":***,"f:InstanceSet/vanilla-kzgbtp-postgresql":***,"f:phase":***,"f:podsReady":***,"f:podsReadyTime":***,"f:conditions":***,"f:observedGeneration":***,"f:phase":***,"subresource":"status"***,***"manager":"kubeblocks","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2025-06-19T11:31:32Z","fieldsType":"FieldsV1","fieldsV1":***"f:metadata":***"f:annotations":***"f:kubeblocks.io/ops-request":***,"f:kubeblocks.io/reconcile":***,"f:finalizers":***".":***,"v:\"cluster.kubeblocks.io/finalizer\"":***,"f:labels":***".":***,"f:clusterdefinition.kubeblocks.io/name":***,"f:clusterversion.kubeblocks.io/name":***,"f:spec":***"f:componentSpecs":***,"f:resources":***".":***,"f:cpu":***,"f:memory":***,"f:services":***,"f:storage":***".":***,"f:size":***]***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","terminationPolicy":"Halt","componentSpecs":[***"name":"postgresql","componentDef":"vanilla-postgresql-14","replicas":2,"resources":***"limits":***"cpu":"600m","memory":"644245094400m"***,"requests":***"cpu":"600m","memory":"644245094400m"***,"volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"6Gi"***]***],"services":[***"name":"postgresql-vpc","serviceName":"postgresql-vpc","annotations":***"service.beta.kubernetes.io/aws-load-balancer-internal":"true","service.beta.kubernetes.io/aws-load-balancer-type":"nlb"***,"spec":***"ports":[***"name":"tcp-postgresql","protocol":"TCP","port":5432,"targetPort":"tcp-postgresql","nodePort":30267***],"type":"LoadBalancer"***,"roleSelector":"primary","componentSelector":"postgresql"***],"resources":***"cpu":"0","memory":"0"***,"storage":***"size":"0"***,"monitor":***,"status":***"observedGeneration":9,"phase":"Failed","components":***"postgresql":***"phase":"Failed","message":***"InstanceSet/vanilla-kzgbtp-postgresql":"[\"vanilla-kzgbtp-postgresql-1\"]"***,"podsReady":false,"podsReadyTime":"2025-06-19T11:15:52Z"***,"clusterDefGeneration":1,"conditions":[***"type":"ProvisioningStarted","status":"True","observedGeneration":9,"lastTransitionTime":"2025-06-19T10:15:26Z","reason":"PreCheckSucceed","message":"The operator has started the provisioning of Cluster: vanilla-kzgbtp"***,***"type":"ApplyResources","status":"True","observedGeneration":9,"lastTransitionTime":"2025-06-19T10:17:01Z","reason":"ApplyResourcesSucceed","message":"Successfully applied for resources"***,***"type":"ReplicasReady","status":"False","lastTransitionTime":"2025-06-19T11:16:07Z","reason":"ReplicasNotReady","message":"pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components"***,***"type":"Ready","status":"False","lastTransitionTime":"2025-06-19T11:16:07Z","reason":"ComponentsNotReady","message":"pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components"***]*** 2025-06-19T11:32:24Z INFO PostgreSQL get local timeline: 9, local lsn: 184549536, isRecovery: true 2025-06-19T11:32:25Z INFO PostgreSQL archive is not enabled panic: start DB service failed: exec command su failed, err:exit status 1, stderr:pg_rewind: fatal: restore_command is not set in the target cluster goroutine 67 [running]: github.com/apecloud/syncer/highavailability.(*Ha).Start(0xc000123340) /src/highavailability/ha.go:257 +0xa53 created by github.com/apecloud/syncer/highavailability.Start in goroutine 1 /src/highavailability/ha.go:69 +0x285 ------------------------------------------------------------------------------------------------------------------ check ops status `kbcli cluster list-ops vanilla-kzgbtp --status all --namespace ns-auxmm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 0/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 0/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 0/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 ops_status:vanilla-kzgbtp-restart-xpvgz ns-auxmm Restart vanilla-kzgbtp postgresql Running 1/2 Jun 19,2025 19:16 UTC+0800 check ops status timeout --------------------------------------get cluster vanilla-kzgbtp yaml-------------------------------------- `kubectl get cluster vanilla-kzgbtp -o yaml --namespace ns-auxmm ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"vanilla-kzgbtp-restart-xpvgz","type":"Restart"***]' kubeblocks.io/reconcile: "2025-06-19T11:41:31.274552093Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"vanilla-kzgbtp","namespace":"ns-auxmm"***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","componentSpecs":[***"name":"postgresql","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"14.7.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"3Gi"***,"storageClassName":null***]***],"terminationPolicy":"Halt","topology":"vanilla-postgresql"*** creationTimestamp: "2025-06-19T10:15:26Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 9 labels: app.kubernetes.io/instance: vanilla-kzgbtp clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" name: vanilla-kzgbtp namespace: ns-auxmm resourceVersion: "87759" uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 spec: clusterDefinitionRef: vanilla-postgresql componentSpecs: - componentDef: vanilla-postgresql-14 name: postgresql replicas: 2 resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m serviceVersion: 14.7.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 6Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: postgresql name: postgresql-vpc roleSelector: primary serviceName: postgresql-vpc spec: ports: - name: tcp-postgresql nodePort: 30267 port: 5432 protocol: TCP targetPort: tcp-postgresql type: LoadBalancer storage: size: "0" terminationPolicy: Halt topology: vanilla-postgresql status: clusterDefGeneration: 1 components: postgresql: message: InstanceSet/vanilla-kzgbtp-postgresql: '["vanilla-kzgbtp-postgresql-1"]' phase: Failed podsReady: false podsReadyTime: "2025-06-19T11:15:52Z" conditions: - lastTransitionTime: "2025-06-19T10:15:26Z" message: 'The operator has started the provisioning of Cluster: vanilla-kzgbtp' observedGeneration: 9 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:17:01Z" message: Successfully applied for resources observedGeneration: 9 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 9 phase: Failed ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster vanilla-kzgbtp-------------------------------------- `kubectl describe cluster vanilla-kzgbtp --namespace ns-auxmm ` Name: vanilla-kzgbtp Namespace: ns-auxmm Labels: app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/ops-request: [***"name":"vanilla-kzgbtp-restart-xpvgz","type":"Restart"***] kubeblocks.io/reconcile: 2025-06-19T11:41:31.274552093Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:26Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 9 Resource Version: 87759 UID: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 Spec: Cluster Definition Ref: vanilla-postgresql Component Specs: Component Def: vanilla-postgresql-14 Name: postgresql Replicas: 2 Resources: Limits: Cpu: 600m Memory: 644245094400m Requests: Cpu: 600m Memory: 644245094400m Service Version: 14.7.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 6Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: postgresql Name: postgresql-vpc Role Selector: primary Service Name: postgresql-vpc Spec: Ports: Name: tcp-postgresql Node Port: 30267 Port: 5432 Protocol: TCP Target Port: tcp-postgresql Type: LoadBalancer Storage: Size: 0 Termination Policy: Halt Topology: vanilla-postgresql Status: Cluster Def Generation: 1 Components: Postgresql: Message: InstanceSet/vanilla-kzgbtp-postgresql: ["vanilla-kzgbtp-postgresql-1"] Phase: Failed Pods Ready: false Pods Ready Time: 2025-06-19T11:15:52Z Conditions: Last Transition Time: 2025-06-19T10:15:26Z Message: The operator has started the provisioning of Cluster: vanilla-kzgbtp Observed Generation: 9 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:17:01Z Message: Successfully applied for resources Observed Generation: 9 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:16:07Z Message: pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:16:07Z Message: pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 9 Phase: Failed Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PreCheckSucceed 42m (x9 over 87m) cluster-controller The operator has started the provisioning of Cluster: vanilla-kzgbtp Warning Unhealthy 27m (x7 over 56m) event-controller Pod vanilla-kzgbtp-postgresql-0: Readiness probe failed: Get "http://172.31.11.240:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal ComponentPhaseTransition 27m (x35 over 87m) cluster-controller component is Running Normal ComponentPhaseTransition 15m (x8 over 67m) cluster-controller component is Failed Normal ComponentPhaseTransition 5m44s (x45 over 85m) cluster-controller component is Updating Warning BackOff 101s (x24 over 26m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) ------------------------------------------------------------------------------------------------------------------ --------------------------------------get opsrequest vanilla-kzgbtp-restart-xpvgz yaml-------------------------------------- `kubectl get opsrequest vanilla-kzgbtp-restart-xpvgz -o yaml --namespace ns-auxmm ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: creationTimestamp: "2025-06-19T11:16:06Z" finalizers: - opsrequest.kubeblocks.io/finalizer generateName: vanilla-kzgbtp-restart- generation: 2 labels: app.kubernetes.io/instance: vanilla-kzgbtp app.kubernetes.io/managed-by: kubeblocks ops.kubeblocks.io/ops-type: Restart name: vanilla-kzgbtp-restart-xpvgz namespace: ns-auxmm ownerReferences: - apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster name: vanilla-kzgbtp uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 resourceVersion: "85053" uid: 3061505b-42e2-48de-8f2f-79bb367d77f5 spec: clusterName: vanilla-kzgbtp enqueueOnForce: false force: true preConditionDeadlineSeconds: 0 restart: - componentName: postgresql type: Restart status: clusterGeneration: 9 components: postgresql: phase: Failed progressDetails: - endTime: "2025-06-19T11:16:21Z" message: 'Failed to restart: Pod/vanilla-kzgbtp-postgresql-1 in Component: postgresql, message: ' objectKey: Pod/vanilla-kzgbtp-postgresql-1 startTime: "2025-06-19T11:16:07Z" status: Failed - objectKey: Pod/vanilla-kzgbtp-postgresql-0 status: Pending conditions: - lastTransitionTime: "2025-06-19T11:16:06Z" message: 'wait for the controller to process the OpsRequest: vanilla-kzgbtp-restart-xpvgz in Cluster: vanilla-kzgbtp' reason: WaitForProgressing status: "True" type: WaitForProgressing - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'OpsRequest: vanilla-kzgbtp-restart-xpvgz is validated' reason: ValidateOpsRequestPassed status: "True" type: Validated - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'Start to restart database in Cluster: vanilla-kzgbtp' reason: RestartStarted status: "True" type: Restarting phase: Running progress: 1/2 startTimestamp: "2025-06-19T11:16:07Z" ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe opsrequest vanilla-kzgbtp-restart-xpvgz-------------------------------------- `kubectl describe opsrequest vanilla-kzgbtp-restart-xpvgz --namespace ns-auxmm ` Name: vanilla-kzgbtp-restart-xpvgz Namespace: ns-auxmm Labels: app.kubernetes.io/instance=vanilla-kzgbtp app.kubernetes.io/managed-by=kubeblocks ops.kubeblocks.io/ops-type=Restart Annotations: API Version: apps.kubeblocks.io/v1alpha1 Kind: OpsRequest Metadata: Creation Timestamp: 2025-06-19T11:16:06Z Finalizers: opsrequest.kubeblocks.io/finalizer Generate Name: vanilla-kzgbtp-restart- Generation: 2 Owner References: API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Name: vanilla-kzgbtp UID: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 Resource Version: 85053 UID: 3061505b-42e2-48de-8f2f-79bb367d77f5 Spec: Cluster Name: vanilla-kzgbtp Enqueue On Force: false Force: true Pre Condition Deadline Seconds: 0 Restart: Component Name: postgresql Type: Restart Status: Cluster Generation: 9 Components: Postgresql: Phase: Failed Progress Details: End Time: 2025-06-19T11:16:21Z Message: Failed to restart: Pod/vanilla-kzgbtp-postgresql-1 in Component: postgresql, message: Object Key: Pod/vanilla-kzgbtp-postgresql-1 Start Time: 2025-06-19T11:16:07Z Status: Failed Object Key: Pod/vanilla-kzgbtp-postgresql-0 Status: Pending Conditions: Last Transition Time: 2025-06-19T11:16:06Z Message: wait for the controller to process the OpsRequest: vanilla-kzgbtp-restart-xpvgz in Cluster: vanilla-kzgbtp Reason: WaitForProgressing Status: True Type: WaitForProgressing Last Transition Time: 2025-06-19T11:16:07Z Message: OpsRequest: vanilla-kzgbtp-restart-xpvgz is validated Reason: ValidateOpsRequestPassed Status: True Type: Validated Last Transition Time: 2025-06-19T11:16:07Z Message: Start to restart database in Cluster: vanilla-kzgbtp Reason: RestartStarted Status: True Type: Restarting Phase: Running Progress: 1/2 Start Timestamp: 2025-06-19T11:16:07Z Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForProgressing 27m ops-request-controller wait for the controller to process the OpsRequest: vanilla-kzgbtp-restart-xpvgz in Cluster: vanilla-kzgbtp Normal ValidateOpsRequestPassed 27m (x2 over 27m) ops-request-controller OpsRequest: vanilla-kzgbtp-restart-xpvgz is validated Normal RestartStarted 27m (x2 over 27m) ops-request-controller Start to restart database in Cluster: vanilla-kzgbtp Normal Processing 27m ops-request-controller Start to restart: Pod/vanilla-kzgbtp-postgresql-1 in Component: postgresql Warning Failed 26m ops-request-controller Failed to restart: Pod/vanilla-kzgbtp-postgresql-1 in Component: postgresql, message: ------------------------------------------------------------------------------------------------------------------ `kubectl get jobs -l ops.kubeblocks.io/ops-name=vanilla-kzgbtp-restart-xpvgz --namespace ns-auxmm` No resources found in ns-auxmm namespace. `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-kzgbtp-restart-xpvgz --namespace ns-auxmm ` opsrequest.apps.kubeblocks.io/vanilla-kzgbtp-restart-xpvgz patched `kbcli cluster delete-ops --name vanilla-kzgbtp-restart-xpvgz --force --auto-approve --namespace ns-auxmm ` OpsRequest vanilla-kzgbtp-restart-xpvgz deleted cluster hscale check cluster status before ops cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-kzgbtp-postgresql-1" force deleted cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-kzgbtp-postgresql-1" force deleted cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-kzgbtp-postgresql-1" force deleted cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed [Error] check cluster status timeout --------------------------------------get cluster vanilla-kzgbtp yaml-------------------------------------- `kubectl get cluster vanilla-kzgbtp -o yaml --namespace ns-auxmm ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T11:51:55.275313042Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"vanilla-kzgbtp","namespace":"ns-auxmm"***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","componentSpecs":[***"name":"postgresql","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"14.7.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"3Gi"***,"storageClassName":null***]***],"terminationPolicy":"Halt","topology":"vanilla-postgresql"*** creationTimestamp: "2025-06-19T10:15:26Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 9 labels: app.kubernetes.io/instance: vanilla-kzgbtp clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" name: vanilla-kzgbtp namespace: ns-auxmm resourceVersion: "95754" uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 spec: clusterDefinitionRef: vanilla-postgresql componentSpecs: - componentDef: vanilla-postgresql-14 name: postgresql replicas: 2 resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m serviceVersion: 14.7.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 6Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: postgresql name: postgresql-vpc roleSelector: primary serviceName: postgresql-vpc spec: ports: - name: tcp-postgresql nodePort: 30267 port: 5432 protocol: TCP targetPort: tcp-postgresql type: LoadBalancer storage: size: "0" terminationPolicy: Halt topology: vanilla-postgresql status: clusterDefGeneration: 1 components: postgresql: message: InstanceSet/vanilla-kzgbtp-postgresql: '["vanilla-kzgbtp-postgresql-1"]' phase: Failed podsReady: false podsReadyTime: "2025-06-19T11:15:52Z" conditions: - lastTransitionTime: "2025-06-19T10:15:26Z" message: 'The operator has started the provisioning of Cluster: vanilla-kzgbtp' observedGeneration: 9 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:17:01Z" message: Successfully applied for resources observedGeneration: 9 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 9 phase: Failed ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster vanilla-kzgbtp-------------------------------------- `kubectl describe cluster vanilla-kzgbtp --namespace ns-auxmm ` Name: vanilla-kzgbtp Namespace: ns-auxmm Labels: app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-06-19T11:51:55.275313042Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:26Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 9 Resource Version: 95754 UID: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 Spec: Cluster Definition Ref: vanilla-postgresql Component Specs: Component Def: vanilla-postgresql-14 Name: postgresql Replicas: 2 Resources: Limits: Cpu: 600m Memory: 644245094400m Requests: Cpu: 600m Memory: 644245094400m Service Version: 14.7.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 6Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: postgresql Name: postgresql-vpc Role Selector: primary Service Name: postgresql-vpc Spec: Ports: Name: tcp-postgresql Node Port: 30267 Port: 5432 Protocol: TCP Target Port: tcp-postgresql Type: LoadBalancer Storage: Size: 0 Termination Policy: Halt Topology: vanilla-postgresql Status: Cluster Def Generation: 1 Components: Postgresql: Message: InstanceSet/vanilla-kzgbtp-postgresql: ["vanilla-kzgbtp-postgresql-1"] Phase: Failed Pods Ready: false Pods Ready Time: 2025-06-19T11:15:52Z Conditions: Last Transition Time: 2025-06-19T10:15:26Z Message: The operator has started the provisioning of Cluster: vanilla-kzgbtp Observed Generation: 9 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:17:01Z Message: Successfully applied for resources Observed Generation: 9 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:16:07Z Message: pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:16:07Z Message: pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 9 Phase: Failed Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PreCheckSucceed 51m (x9 over 96m) cluster-controller The operator has started the provisioning of Cluster: vanilla-kzgbtp Warning Unhealthy 36m (x7 over 65m) event-controller Pod vanilla-kzgbtp-postgresql-0: Readiness probe failed: Get "http://172.31.11.240:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal ComponentPhaseTransition 36m (x35 over 95m) cluster-controller component is Running Warning BackOff 10m (x24 over 35m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) Normal ComponentPhaseTransition 6m4s (x11 over 76m) cluster-controller component is Failed Warning BackOff 5m33s event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(83f5f37c-dc51-4f94-ba4d-e6fb922ad488) Warning BackOff 2m20s (x4 over 2m56s) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(852b6fa1-e02c-4a09-8631-bbe0cf86840d) Normal ComponentPhaseTransition 77s (x53 over 94m) cluster-controller component is Updating ------------------------------------------------------------------------------------------------------------------ cluster hscale check cluster status before ops cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-kzgbtp-postgresql-1" force deleted cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-kzgbtp-postgresql-1" force deleted cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-kzgbtp-postgresql-1" force deleted cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed [Error] check cluster status timeout --------------------------------------get cluster vanilla-kzgbtp yaml-------------------------------------- `kubectl get cluster vanilla-kzgbtp -o yaml --namespace ns-auxmm ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T12:00:42.283326792Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"vanilla-kzgbtp","namespace":"ns-auxmm"***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","componentSpecs":[***"name":"postgresql","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"14.7.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"3Gi"***,"storageClassName":null***]***],"terminationPolicy":"Halt","topology":"vanilla-postgresql"*** creationTimestamp: "2025-06-19T10:15:26Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 9 labels: app.kubernetes.io/instance: vanilla-kzgbtp clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" name: vanilla-kzgbtp namespace: ns-auxmm resourceVersion: "102766" uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 spec: clusterDefinitionRef: vanilla-postgresql componentSpecs: - componentDef: vanilla-postgresql-14 name: postgresql replicas: 2 resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m serviceVersion: 14.7.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 6Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: postgresql name: postgresql-vpc roleSelector: primary serviceName: postgresql-vpc spec: ports: - name: tcp-postgresql nodePort: 30267 port: 5432 protocol: TCP targetPort: tcp-postgresql type: LoadBalancer storage: size: "0" terminationPolicy: Halt topology: vanilla-postgresql status: clusterDefGeneration: 1 components: postgresql: message: InstanceSet/vanilla-kzgbtp-postgresql: '["vanilla-kzgbtp-postgresql-1"]' phase: Failed podsReady: false podsReadyTime: "2025-06-19T11:15:52Z" conditions: - lastTransitionTime: "2025-06-19T10:15:26Z" message: 'The operator has started the provisioning of Cluster: vanilla-kzgbtp' observedGeneration: 9 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:17:01Z" message: Successfully applied for resources observedGeneration: 9 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 9 phase: Failed ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster vanilla-kzgbtp-------------------------------------- `kubectl describe cluster vanilla-kzgbtp --namespace ns-auxmm ` Name: vanilla-kzgbtp Namespace: ns-auxmm Labels: app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-06-19T12:00:42.283326792Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:26Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 9 Resource Version: 102766 UID: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 Spec: Cluster Definition Ref: vanilla-postgresql Component Specs: Component Def: vanilla-postgresql-14 Name: postgresql Replicas: 2 Resources: Limits: Cpu: 600m Memory: 644245094400m Requests: Cpu: 600m Memory: 644245094400m Service Version: 14.7.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 6Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: postgresql Name: postgresql-vpc Role Selector: primary Service Name: postgresql-vpc Spec: Ports: Name: tcp-postgresql Node Port: 30267 Port: 5432 Protocol: TCP Target Port: tcp-postgresql Type: LoadBalancer Storage: Size: 0 Termination Policy: Halt Topology: vanilla-postgresql Status: Cluster Def Generation: 1 Components: Postgresql: Message: InstanceSet/vanilla-kzgbtp-postgresql: ["vanilla-kzgbtp-postgresql-1"] Phase: Failed Pods Ready: false Pods Ready Time: 2025-06-19T11:15:52Z Conditions: Last Transition Time: 2025-06-19T10:15:26Z Message: The operator has started the provisioning of Cluster: vanilla-kzgbtp Observed Generation: 9 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:17:01Z Message: Successfully applied for resources Observed Generation: 9 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:16:07Z Message: pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:16:07Z Message: pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 9 Phase: Failed Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PreCheckSucceed 60m (x9 over 105m) cluster-controller The operator has started the provisioning of Cluster: vanilla-kzgbtp Warning Unhealthy 45m (x7 over 73m) event-controller Pod vanilla-kzgbtp-postgresql-0: Readiness probe failed: Get "http://172.31.11.240:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal ComponentPhaseTransition 45m (x35 over 104m) cluster-controller component is Running Warning BackOff 19m (x24 over 43m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) Warning BackOff 14m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(83f5f37c-dc51-4f94-ba4d-e6fb922ad488) Warning BackOff 11m (x4 over 11m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(852b6fa1-e02c-4a09-8631-bbe0cf86840d) Warning BackOff 5m31s event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(abc1339b-784d-43ec-b94a-45fa3e555546) Normal ComponentPhaseTransition 5m18s (x59 over 103m) cluster-controller component is Updating Warning BackOff 36s event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(b4f352df-30fb-4301-a14b-3114c1c8b565) Normal ComponentPhaseTransition 6s (x32 over 85m) cluster-controller component is Failed ------------------------------------------------------------------------------------------------------------------ test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-vanilla-kzgbtp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-vanilla-kzgbtp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-vanilla-kzgbtp namespace: ns-auxmm spec: selector: namespaces: - ns-auxmm labelSelectors: apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-vanilla-kzgbtp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-vanilla-kzgbtp created apply test-chaos-mesh-networkduplicate-vanilla-kzgbtp.yaml Success `rm -rf test-chaos-mesh-networkduplicate-vanilla-kzgbtp.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql Halt Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-kzgbtp-postgresql-1" force deleted cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-kzgbtp-postgresql-1" force deleted cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-kzgbtp-postgresql-1" force deleted cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed [Error] check cluster status timeout --------------------------------------get cluster vanilla-kzgbtp yaml-------------------------------------- `kubectl get cluster vanilla-kzgbtp -o yaml --namespace ns-auxmm ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T12:11:35.273555534Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"vanilla-kzgbtp","namespace":"ns-auxmm"***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","componentSpecs":[***"name":"postgresql","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"14.7.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"3Gi"***,"storageClassName":null***]***],"terminationPolicy":"Halt","topology":"vanilla-postgresql"*** creationTimestamp: "2025-06-19T10:15:26Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 9 labels: app.kubernetes.io/instance: vanilla-kzgbtp clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" name: vanilla-kzgbtp namespace: ns-auxmm resourceVersion: "109089" uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 spec: clusterDefinitionRef: vanilla-postgresql componentSpecs: - componentDef: vanilla-postgresql-14 name: postgresql replicas: 2 resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m serviceVersion: 14.7.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 6Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: postgresql name: postgresql-vpc roleSelector: primary serviceName: postgresql-vpc spec: ports: - name: tcp-postgresql nodePort: 30267 port: 5432 protocol: TCP targetPort: tcp-postgresql type: LoadBalancer storage: size: "0" terminationPolicy: Halt topology: vanilla-postgresql status: clusterDefGeneration: 1 components: postgresql: message: InstanceSet/vanilla-kzgbtp-postgresql: '["vanilla-kzgbtp-postgresql-1"]' phase: Failed podsReady: false podsReadyTime: "2025-06-19T11:15:52Z" conditions: - lastTransitionTime: "2025-06-19T10:15:26Z" message: 'The operator has started the provisioning of Cluster: vanilla-kzgbtp' observedGeneration: 9 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:17:01Z" message: Successfully applied for resources observedGeneration: 9 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 9 phase: Failed ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster vanilla-kzgbtp-------------------------------------- `kubectl describe cluster vanilla-kzgbtp --namespace ns-auxmm ` Name: vanilla-kzgbtp Namespace: ns-auxmm Labels: app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-06-19T12:11:35.273555534Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:26Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 9 Resource Version: 109089 UID: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 Spec: Cluster Definition Ref: vanilla-postgresql Component Specs: Component Def: vanilla-postgresql-14 Name: postgresql Replicas: 2 Resources: Limits: Cpu: 600m Memory: 644245094400m Requests: Cpu: 600m Memory: 644245094400m Service Version: 14.7.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 6Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: postgresql Name: postgresql-vpc Role Selector: primary Service Name: postgresql-vpc Spec: Ports: Name: tcp-postgresql Node Port: 30267 Port: 5432 Protocol: TCP Target Port: tcp-postgresql Type: LoadBalancer Storage: Size: 0 Termination Policy: Halt Topology: vanilla-postgresql Status: Cluster Def Generation: 1 Components: Postgresql: Message: InstanceSet/vanilla-kzgbtp-postgresql: ["vanilla-kzgbtp-postgresql-1"] Phase: Failed Pods Ready: false Pods Ready Time: 2025-06-19T11:15:52Z Conditions: Last Transition Time: 2025-06-19T10:15:26Z Message: The operator has started the provisioning of Cluster: vanilla-kzgbtp Observed Generation: 9 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:17:01Z Message: Successfully applied for resources Observed Generation: 9 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:16:07Z Message: pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:16:07Z Message: pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 9 Phase: Failed Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 56m (x7 over 84m) event-controller Pod vanilla-kzgbtp-postgresql-0: Readiness probe failed: Get "http://172.31.11.240:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal ComponentPhaseTransition 55m (x35 over 115m) cluster-controller component is Running Warning BackOff 30m (x24 over 54m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) Warning BackOff 25m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(83f5f37c-dc51-4f94-ba4d-e6fb922ad488) Warning BackOff 21m (x4 over 22m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(852b6fa1-e02c-4a09-8631-bbe0cf86840d) Warning BackOff 16m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(abc1339b-784d-43ec-b94a-45fa3e555546) Normal ComponentPhaseTransition 16m (x59 over 114m) cluster-controller component is Updating Warning BackOff 11m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(b4f352df-30fb-4301-a14b-3114c1c8b565) Warning BackOff 5m33s event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(de616e09-59b6-4dfc-8f48-61db6dd9919b) Warning BackOff 115s (x6 over 3m4s) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(9824e84b-e29c-49dd-afda-18d64c6948b2) Normal ComponentPhaseTransition 65s (x43 over 96m) cluster-controller component is Failed ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql CrashLoopBackOff us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 20:10 UTC+0800 pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:Error pod_status:Error pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:Error pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff check pod status done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-vanilla-kzgbtp --namespace ns-auxmm ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-vanilla-kzgbtp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-vanilla-kzgbtp" not found check failover pod name failover pod name:vanilla-kzgbtp-postgresql-0 failover networkduplicate Success cluster update terminationPolicy WipeOut `kbcli cluster update vanilla-kzgbtp --termination-policy=WipeOut --namespace ns-auxmm ` cluster.apps.kubeblocks.io/vanilla-kzgbtp updated check cluster status `kbcli cluster list vanilla-kzgbtp --show-labels --namespace ns-auxmm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-kzgbtp ns-auxmm vanilla-postgresql WipeOut Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=vanilla-kzgbtp,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed [Error] check cluster status timeout --------------------------------------get cluster vanilla-kzgbtp yaml-------------------------------------- `kubectl get cluster vanilla-kzgbtp -o yaml --namespace ns-auxmm ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T12:20:38.29178368Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"vanilla-kzgbtp","namespace":"ns-auxmm"***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","componentSpecs":[***"name":"postgresql","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"14.7.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"3Gi"***,"storageClassName":null***]***],"terminationPolicy":"Halt","topology":"vanilla-postgresql"*** creationTimestamp: "2025-06-19T10:15:26Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 10 labels: app.kubernetes.io/instance: vanilla-kzgbtp clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" name: vanilla-kzgbtp namespace: ns-auxmm resourceVersion: "114526" uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 spec: clusterDefinitionRef: vanilla-postgresql componentSpecs: - componentDef: vanilla-postgresql-14 name: postgresql replicas: 2 resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m serviceVersion: 14.7.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 6Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: postgresql name: postgresql-vpc roleSelector: primary serviceName: postgresql-vpc spec: ports: - name: tcp-postgresql nodePort: 30267 port: 5432 protocol: TCP targetPort: tcp-postgresql type: LoadBalancer storage: size: "0" terminationPolicy: WipeOut topology: vanilla-postgresql status: clusterDefGeneration: 1 components: postgresql: message: InstanceSet/vanilla-kzgbtp-postgresql: '["vanilla-kzgbtp-postgresql-1"]' phase: Failed podsReady: false podsReadyTime: "2025-06-19T11:15:52Z" conditions: - lastTransitionTime: "2025-06-19T10:15:26Z" message: 'The operator has started the provisioning of Cluster: vanilla-kzgbtp' observedGeneration: 10 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:17:01Z" message: Successfully applied for resources observedGeneration: 10 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 10 phase: Failed ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster vanilla-kzgbtp-------------------------------------- `kubectl describe cluster vanilla-kzgbtp --namespace ns-auxmm ` Name: vanilla-kzgbtp Namespace: ns-auxmm Labels: app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-06-19T12:20:38.29178368Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:26Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 10 Resource Version: 114526 UID: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 Spec: Cluster Definition Ref: vanilla-postgresql Component Specs: Component Def: vanilla-postgresql-14 Name: postgresql Replicas: 2 Resources: Limits: Cpu: 600m Memory: 644245094400m Requests: Cpu: 600m Memory: 644245094400m Service Version: 14.7.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 6Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: postgresql Name: postgresql-vpc Role Selector: primary Service Name: postgresql-vpc Spec: Ports: Name: tcp-postgresql Node Port: 30267 Port: 5432 Protocol: TCP Target Port: tcp-postgresql Type: LoadBalancer Storage: Size: 0 Termination Policy: WipeOut Topology: vanilla-postgresql Status: Cluster Def Generation: 1 Components: Postgresql: Message: InstanceSet/vanilla-kzgbtp-postgresql: ["vanilla-kzgbtp-postgresql-1"] Phase: Failed Pods Ready: false Pods Ready Time: 2025-06-19T11:15:52Z Conditions: Last Transition Time: 2025-06-19T10:15:26Z Message: The operator has started the provisioning of Cluster: vanilla-kzgbtp Observed Generation: 10 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:17:01Z Message: Successfully applied for resources Observed Generation: 10 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:16:07Z Message: pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:16:07Z Message: pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 10 Phase: Failed Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 43m (x24 over 68m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) Warning BackOff 38m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(83f5f37c-dc51-4f94-ba4d-e6fb922ad488) Warning BackOff 35m (x4 over 35m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(852b6fa1-e02c-4a09-8631-bbe0cf86840d) Warning BackOff 29m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(abc1339b-784d-43ec-b94a-45fa3e555546) Warning BackOff 24m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(b4f352df-30fb-4301-a14b-3114c1c8b565) Warning BackOff 18m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(de616e09-59b6-4dfc-8f48-61db6dd9919b) Warning BackOff 15m (x6 over 16m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(9824e84b-e29c-49dd-afda-18d64c6948b2) Normal ComponentPhaseTransition 14m (x43 over 109m) cluster-controller component is Failed Warning BackOff 4m17s (x20 over 13m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(e11bcf51-ea66-478a-995a-2dc548b6c9e5) Normal ComponentPhaseTransition 3m42s (x84 over 127m) cluster-controller component is Updating ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances vanilla-kzgbtp --namespace ns-auxmm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-kzgbtp-postgresql-0 ns-auxmm vanilla-kzgbtp postgresql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Jun 19,2025 18:31 UTC+0800 vanilla-kzgbtp-postgresql-1 ns-auxmm vanilla-kzgbtp postgresql CrashLoopBackOff us-west-2a 600m / 600m 644245094400m / 644245094400m data:6Gi ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Jun 19,2025 20:10 UTC+0800 pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:Error pod_status:Error pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:Error pod_status:Error pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff [Error] check pod status timeout --------------------------------------get cluster vanilla-kzgbtp yaml-------------------------------------- `kubectl get cluster vanilla-kzgbtp -o yaml --namespace ns-auxmm ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T12:30:41.285628969Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"vanilla-kzgbtp","namespace":"ns-auxmm"***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","componentSpecs":[***"name":"postgresql","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"14.7.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"3Gi"***,"storageClassName":null***]***],"terminationPolicy":"Halt","topology":"vanilla-postgresql"*** creationTimestamp: "2025-06-19T10:15:26Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 10 labels: app.kubernetes.io/instance: vanilla-kzgbtp clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" name: vanilla-kzgbtp namespace: ns-auxmm resourceVersion: "120054" uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 spec: clusterDefinitionRef: vanilla-postgresql componentSpecs: - componentDef: vanilla-postgresql-14 name: postgresql replicas: 2 resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m serviceVersion: 14.7.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 6Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: postgresql name: postgresql-vpc roleSelector: primary serviceName: postgresql-vpc spec: ports: - name: tcp-postgresql nodePort: 30267 port: 5432 protocol: TCP targetPort: tcp-postgresql type: LoadBalancer storage: size: "0" terminationPolicy: WipeOut topology: vanilla-postgresql status: clusterDefGeneration: 1 components: postgresql: message: InstanceSet/vanilla-kzgbtp-postgresql: '["vanilla-kzgbtp-postgresql-1"]' phase: Failed podsReady: false podsReadyTime: "2025-06-19T11:15:52Z" conditions: - lastTransitionTime: "2025-06-19T10:15:26Z" message: 'The operator has started the provisioning of Cluster: vanilla-kzgbtp' observedGeneration: 10 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:17:01Z" message: Successfully applied for resources observedGeneration: 10 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:16:07Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 10 phase: Failed ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster vanilla-kzgbtp-------------------------------------- `kubectl describe cluster vanilla-kzgbtp --namespace ns-auxmm ` Name: vanilla-kzgbtp Namespace: ns-auxmm Labels: app.kubernetes.io/instance=vanilla-kzgbtp clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-06-19T12:30:41.285628969Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:26Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 10 Resource Version: 120054 UID: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 Spec: Cluster Definition Ref: vanilla-postgresql Component Specs: Component Def: vanilla-postgresql-14 Name: postgresql Replicas: 2 Resources: Limits: Cpu: 600m Memory: 644245094400m Requests: Cpu: 600m Memory: 644245094400m Service Version: 14.7.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 6Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: postgresql Name: postgresql-vpc Role Selector: primary Service Name: postgresql-vpc Spec: Ports: Name: tcp-postgresql Node Port: 30267 Port: 5432 Protocol: TCP Target Port: tcp-postgresql Type: LoadBalancer Storage: Size: 0 Termination Policy: WipeOut Topology: vanilla-postgresql Status: Cluster Def Generation: 1 Components: Postgresql: Message: InstanceSet/vanilla-kzgbtp-postgresql: ["vanilla-kzgbtp-postgresql-1"] Phase: Failed Pods Ready: false Pods Ready Time: 2025-06-19T11:15:52Z Conditions: Last Transition Time: 2025-06-19T10:15:26Z Message: The operator has started the provisioning of Cluster: vanilla-kzgbtp Observed Generation: 10 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:17:01Z Message: Successfully applied for resources Observed Generation: 10 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:16:07Z Message: pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:16:07Z Message: pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 10 Phase: Failed Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 52m (x24 over 77m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(f6124d06-79c8-478c-9124-c3bb4a090337) Warning BackOff 47m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(83f5f37c-dc51-4f94-ba4d-e6fb922ad488) Warning BackOff 44m (x4 over 45m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(852b6fa1-e02c-4a09-8631-bbe0cf86840d) Warning BackOff 39m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(abc1339b-784d-43ec-b94a-45fa3e555546) Warning BackOff 34m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(b4f352df-30fb-4301-a14b-3114c1c8b565) Warning BackOff 28m event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(de616e09-59b6-4dfc-8f48-61db6dd9919b) Warning BackOff 24m (x6 over 25m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(9824e84b-e29c-49dd-afda-18d64c6948b2) Normal ComponentPhaseTransition 23m (x43 over 118m) cluster-controller component is Failed Warning BackOff 3m47s (x22 over 23m) event-controller Pod vanilla-kzgbtp-postgresql-1: Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(e11bcf51-ea66-478a-995a-2dc548b6c9e5) Normal ComponentPhaseTransition 3m4s (x86 over 136m) cluster-controller component is Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------get pod vanilla-kzgbtp-postgresql-0 vanilla-kzgbtp-postgresql-1 yaml-------------------------------------- `kubectl get pod vanilla-kzgbtp-postgresql-0 -o yaml --namespace ns-auxmm ` apiVersion: v1 kind: Pod metadata: annotations: apps.kubeblocks.io/component-replicas: "2" apps.kubeblocks.io/last-role-snapshot-version: "1750329814096971" creationTimestamp: "2025-06-19T10:31:40Z" labels: app.kubernetes.io/component: vanilla-postgresql-14 app.kubernetes.io/instance: vanilla-kzgbtp app.kubernetes.io/managed-by: kubeblocks app.kubernetes.io/name: vanilla-postgresql-14 app.kubernetes.io/version: vanilla-postgresql-14 apps.kubeblocks.io/cluster-uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 apps.kubeblocks.io/component-name: postgresql apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-0 clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" componentdefinition.kubeblocks.io/name: vanilla-postgresql-14 controller-revision-hash: 6c8d9d7bf5 kubeblocks.io/role: primary workloads.kubeblocks.io/access-mode: ReadWrite workloads.kubeblocks.io/instance: vanilla-kzgbtp-postgresql workloads.kubeblocks.io/managed-by: InstanceSet name: vanilla-kzgbtp-postgresql-0 namespace: ns-auxmm ownerReferences: - apiVersion: workloads.kubeblocks.io/v1alpha1 blockOwnerDeletion: true controller: true kind: InstanceSet name: vanilla-kzgbtp-postgresql uid: 7be55aad-9aa7-48fb-b705-0b349c86f2aa resourceVersion: "69281" uid: f3b3a75d-ecb7-4978-baeb-a7597f9d8a14 spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: kb-data operator: In values: - "true" weight: 100 containers: - command: - /kubeblocks/syncer - --port - "3601" - -- - docker-entrypoint.sh - --config-file=/postgresql/volume_data/conf/postgresql.conf - --hba_file=/postgresql/volume_data/conf/pg_hba.conf env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: ALLOW_NOSSL value: "true" - name: POSTGRESQL_PORT_NUMBER value: "5432" - name: PGDATA value: /postgresql/volume_data/data - name: PGCONF value: /postgresql/volume_data/conf - name: POSTGRESQL_MOUNTED_CONF_DIR value: /postgresql/mount_conf - name: PGUSER value: $(POSTGRES_USER) - name: PGPASSWORD value: $(POSTGRES_PASSWORD) - name: KB_ENGINE_TYPE value: vanilla-postgresql envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/postgres:14.7 imagePullPolicy: IfNotPresent name: postgresql ports: - containerPort: 5432 name: tcp-postgresql protocol: TCP resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m securityContext: runAsUser: 0 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /postgresql/volume_data name: data - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /kb-scripts name: scripts - mountPath: /kubeblocks name: kubeblocks - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true - command: - lorry - --port - "3501" - --grpcport - "50001" env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: KB_RSM_ROLE_PROBE_PERIOD value: "1" - name: KB_BUILTIN_HANDLER value: postgresql - name: KB_SERVICE_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_SERVICE_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_SERVICE_PORT value: "5432" - name: KB_DATA_PATH value: /postgresql/volume_data - name: KB_RSM_ACTION_SVC_LIST value: "null" - name: KB_RSM_ROLE_UPDATE_MECHANISM value: DirectAPIServerEventUpdate - name: KB_RSM_ROLE_PROBE_TIMEOUT value: "1" - name: KB_CLUSTER_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.labels['app.kubernetes.io/instance'] - name: KB_COMP_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.labels['apps.kubeblocks.io/component-name'] - name: KB_SERVICE_CHARACTER_TYPE value: postgresql envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/kubeblocks-tools:0.9.4 imagePullPolicy: IfNotPresent name: lorry ports: - containerPort: 3501 name: lorry-http-port protocol: TCP - containerPort: 50001 name: lorry-grpc-port protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /v1.0/checkrole port: 3501 scheme: HTTP periodSeconds: 1 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" startupProbe: failureThreshold: 3 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 3501 timeoutSeconds: 1 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/volume_data name: data - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true - args: - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) - /kb_tools/reloader - --log-level - info - --operator-update-enable - --tcp - "9901" - --config - /opt/config-manager/config-manager.yaml command: - env env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: CONFIG_MANAGER_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: TOOLS_PATH value: /opt/kb-tools/reload/vanilla-postgresql-configuration:/opt/config-manager:/kb_tools envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/postgres:14.7 imagePullPolicy: IfNotPresent name: config-manager ports: - containerPort: 9901 name: config-manager protocol: TCP resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostname: vanilla-kzgbtp-postgresql-0 initContainers: - command: - sh - -c - cp -r /bin/syncer /kubeblocks/ env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false image: docker.io/apecloud/syncer:0.3.6 imagePullPolicy: IfNotPresent name: init-syncer resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /kubeblocks name: kubeblocks - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true - command: - cp - /bin/reloader - /kb_tools env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false image: docker.io/apecloud/kubeblocks-tools:0.9.4 imagePullPolicy: IfNotPresent name: install-config-manager-tool resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true nodeName: ip-172-31-6-29.us-west-2.compute.internal preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 103 runAsGroup: 103 runAsUser: 0 serviceAccount: kb-vanilla-kzgbtp serviceAccountName: kb-vanilla-kzgbtp subdomain: vanilla-kzgbtp-postgresql-headless terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule key: kb-data operator: Equal value: "true" - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - emptyDir: medium: Memory sizeLimit: 644245094400m name: dshm - configMap: defaultMode: 292 name: vanilla-kzgbtp-postgresql-vanilla-postgresql-configuration name: postgresql-config - configMap: defaultMode: 365 name: vanilla-kzgbtp-postgresql-vanilla-postgresql-scripts name: scripts - configMap: defaultMode: 493 name: sidecar-vanilla-postgresql-reload-tools-script-vanilla-kzgbtp name: cm-script-vanilla-postgresql-configuration - configMap: defaultMode: 493 name: sidecar-vanilla-kzgbtp-postgresql-config-manager-config name: config-manager-config - emptyDir: *** name: kb-tools - name: data persistentVolumeClaim: claimName: data-vanilla-kzgbtp-postgresql-0 - emptyDir: *** name: kubeblocks - name: kube-api-access-lfdrn projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2025-06-19T10:31:45Z" status: "True" type: PodReadyToStartContainers - lastProbeTime: null lastTransitionTime: "2025-06-19T10:31:46Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2025-06-19T11:15:51Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2025-06-19T11:15:51Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2025-06-19T10:31:40Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://4a559d5430038eadb6ec4f2b6654901ec74628489218239605b3f1d1ae551206 image: docker.io/apecloud/postgres:14.7 imageID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c lastState: terminated: containerID: containerd://c4ff4251b5dc1f845c9cc8eacab8d2f149fa4c757d26c6a9c3db59a4e75a900f exitCode: 128 finishedAt: "2025-06-19T10:41:16Z" message: 'failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "env": executable file not found in $PATH: unknown' reason: StartError startedAt: "1970-01-01T00:00:00Z" name: config-manager ready: true restartCount: 6 started: true state: running: startedAt: "2025-06-19T10:41:37Z" volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled - containerID: containerd://fa38d6cf08bb686721e3f36bc4e802c5b83c45ea15d8c4b62134e4f1153a1a6d image: docker.io/apecloud/kubeblocks-tools:0.9.4 imageID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 lastState: terminated: containerID: containerd://e5fb6dd866135c11e8bf4dd6c9f1ac111a74578636f1ee6eb98f56360c34a7fa exitCode: 128 finishedAt: "2025-06-19T10:41:15Z" message: 'failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "lorry": executable file not found in $PATH: unknown' reason: StartError startedAt: "1970-01-01T00:00:00Z" name: lorry ready: true restartCount: 6 started: true state: running: startedAt: "2025-06-19T10:41:37Z" volumeMounts: - mountPath: /postgresql/volume_data name: data - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled - containerID: containerd://fe46f97f33678fa6ae8cf97d6d4e3035f3a031c1935e8e3a6e427d99071bf81e image: docker.io/apecloud/postgres:14.7 imageID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c lastState: terminated: containerID: containerd://820956b0b3afbf9bba2d6074479311619ab1eef40d8bc102418383e8e014485f exitCode: 2 finishedAt: "2025-06-19T10:41:15Z" reason: Error startedAt: "2025-06-19T10:41:15Z" name: postgresql ready: true restartCount: 7 started: true state: running: startedAt: "2025-06-19T10:41:37Z" volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /postgresql/volume_data name: data - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /kb-scripts name: scripts - mountPath: /kubeblocks name: kubeblocks - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled hostIP: 172.31.6.29 hostIPs: - ip: 172.31.6.29 initContainerStatuses: - containerID: containerd://5e5278ee7a8be5bdf9eef18a2c7b8cc17cdece0282eb90c489143095bc5c5e9f image: docker.io/apecloud/syncer:0.3.6 imageID: docker.io/apecloud/syncer@sha256:d8cb38ff8f9e025e6211725bb61ac1cbaadb10dc5e9dc6a2613ef09f7a42f26e lastState: *** name: init-syncer ready: true restartCount: 0 started: false state: terminated: containerID: containerd://5e5278ee7a8be5bdf9eef18a2c7b8cc17cdece0282eb90c489143095bc5c5e9f exitCode: 0 finishedAt: "2025-06-19T10:31:44Z" reason: Completed startedAt: "2025-06-19T10:31:44Z" volumeMounts: - mountPath: /kubeblocks name: kubeblocks - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled - containerID: containerd://4d462f4a83b1519c07e61de32cee6a50957374b57f9975eceeec7723ab1fa807 image: docker.io/apecloud/kubeblocks-tools:0.9.4 imageID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 lastState: *** name: install-config-manager-tool ready: true restartCount: 0 started: false state: terminated: containerID: containerd://4d462f4a83b1519c07e61de32cee6a50957374b57f9975eceeec7723ab1fa807 exitCode: 0 finishedAt: "2025-06-19T10:31:45Z" reason: Completed startedAt: "2025-06-19T10:31:45Z" volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-lfdrn readOnly: true recursiveReadOnly: Disabled phase: Running podIP: 172.31.11.240 podIPs: - ip: 172.31.11.240 qosClass: Burstable startTime: "2025-06-19T10:31:40Z" ------------------------------------------------------------------------------------------------------------------ `kubectl get pod vanilla-kzgbtp-postgresql-1 -o yaml --namespace ns-auxmm ` apiVersion: v1 kind: Pod metadata: annotations: apps.kubeblocks.io/component-replicas: "2" kubeblocks.io/restart: "2025-06-19T11:16:07Z" creationTimestamp: "2025-06-19T12:10:22Z" labels: app.kubernetes.io/component: vanilla-postgresql-14 app.kubernetes.io/instance: vanilla-kzgbtp app.kubernetes.io/managed-by: kubeblocks app.kubernetes.io/name: vanilla-postgresql-14 app.kubernetes.io/version: vanilla-postgresql-14 apps.kubeblocks.io/cluster-uid: 9fe00cca-f01b-4f4e-8e1c-f33f8591d154 apps.kubeblocks.io/component-name: postgresql apps.kubeblocks.io/pod-name: vanilla-kzgbtp-postgresql-1 clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" componentdefinition.kubeblocks.io/name: vanilla-postgresql-14 controller-revision-hash: 6bfcc759f workloads.kubeblocks.io/instance: vanilla-kzgbtp-postgresql workloads.kubeblocks.io/managed-by: InstanceSet name: vanilla-kzgbtp-postgresql-1 namespace: ns-auxmm ownerReferences: - apiVersion: workloads.kubeblocks.io/v1alpha1 blockOwnerDeletion: true controller: true kind: InstanceSet name: vanilla-kzgbtp-postgresql uid: 7be55aad-9aa7-48fb-b705-0b349c86f2aa resourceVersion: "120049" uid: e11bcf51-ea66-478a-995a-2dc548b6c9e5 spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: kb-data operator: In values: - "true" weight: 100 containers: - command: - /kubeblocks/syncer - --port - "3601" - -- - docker-entrypoint.sh - --config-file=/postgresql/volume_data/conf/postgresql.conf - --hba_file=/postgresql/volume_data/conf/pg_hba.conf env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: ALLOW_NOSSL value: "true" - name: POSTGRESQL_PORT_NUMBER value: "5432" - name: PGDATA value: /postgresql/volume_data/data - name: PGCONF value: /postgresql/volume_data/conf - name: POSTGRESQL_MOUNTED_CONF_DIR value: /postgresql/mount_conf - name: PGUSER value: $(POSTGRES_USER) - name: PGPASSWORD value: $(POSTGRES_PASSWORD) - name: KB_ENGINE_TYPE value: vanilla-postgresql envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/postgres:14.7 imagePullPolicy: IfNotPresent name: postgresql ports: - containerPort: 5432 name: tcp-postgresql protocol: TCP resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m securityContext: runAsUser: 0 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /postgresql/volume_data name: data - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /kb-scripts name: scripts - mountPath: /kubeblocks name: kubeblocks - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true - command: - lorry - --port - "3501" - --grpcport - "50001" env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: KB_RSM_ROLE_PROBE_PERIOD value: "1" - name: KB_BUILTIN_HANDLER value: postgresql - name: KB_SERVICE_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_SERVICE_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_SERVICE_PORT value: "5432" - name: KB_DATA_PATH value: /postgresql/volume_data - name: KB_RSM_ACTION_SVC_LIST value: "null" - name: KB_RSM_ROLE_UPDATE_MECHANISM value: DirectAPIServerEventUpdate - name: KB_RSM_ROLE_PROBE_TIMEOUT value: "1" - name: KB_CLUSTER_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.labels['app.kubernetes.io/instance'] - name: KB_COMP_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.labels['apps.kubeblocks.io/component-name'] - name: KB_SERVICE_CHARACTER_TYPE value: postgresql envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/kubeblocks-tools:0.9.4 imagePullPolicy: IfNotPresent name: lorry ports: - containerPort: 3501 name: lorry-http-port protocol: TCP - containerPort: 50001 name: lorry-grpc-port protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /v1.0/checkrole port: 3501 scheme: HTTP periodSeconds: 1 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" startupProbe: failureThreshold: 3 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 3501 timeoutSeconds: 1 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/volume_data name: data - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true - args: - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) - /kb_tools/reloader - --log-level - info - --operator-update-enable - --tcp - "9901" - --config - /opt/config-manager/config-manager.yaml command: - env env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc - name: CONFIG_MANAGER_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: TOOLS_PATH value: /opt/kb-tools/reload/vanilla-postgresql-configuration:/opt/config-manager:/kb_tools envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false - configMapRef: name: vanilla-kzgbtp-postgresql-rsm-env optional: false image: docker.io/apecloud/postgres:14.7 imagePullPolicy: IfNotPresent name: config-manager ports: - containerPort: 9901 name: config-manager protocol: TCP resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostname: vanilla-kzgbtp-postgresql-1 initContainers: - command: - sh - -c - cp -r /bin/syncer /kubeblocks/ env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false image: docker.io/apecloud/syncer:0.3.6 imagePullPolicy: IfNotPresent name: init-syncer resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /kubeblocks name: kubeblocks - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true - command: - cp - /bin/reloader - /kb_tools env: - name: POSTGRES_USER valueFrom: secretKeyRef: key: username name: vanilla-kzgbtp-postgresql-account-postgres - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: vanilla-kzgbtp-postgresql-account-postgres - name: KB_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KB_POD_UID valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.uid - name: KB_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KB_SA_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.serviceAccountName - name: KB_NODENAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: KB_HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_POD_IPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_HOSTIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: KB_PODIP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: KB_PODIPS valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIPs - name: KB_POD_FQDN value: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc envFrom: - configMapRef: name: vanilla-kzgbtp-postgresql-env optional: false image: docker.io/apecloud/kubeblocks-tools:0.9.4 imagePullPolicy: IfNotPresent name: install-config-manager-tool resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true nodeName: ip-172-31-14-226.us-west-2.compute.internal preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 103 runAsGroup: 103 runAsUser: 0 serviceAccount: kb-vanilla-kzgbtp serviceAccountName: kb-vanilla-kzgbtp subdomain: vanilla-kzgbtp-postgresql-headless terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule key: kb-data operator: Equal value: "true" - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - emptyDir: medium: Memory sizeLimit: 644245094400m name: dshm - configMap: defaultMode: 292 name: vanilla-kzgbtp-postgresql-vanilla-postgresql-configuration name: postgresql-config - configMap: defaultMode: 365 name: vanilla-kzgbtp-postgresql-vanilla-postgresql-scripts name: scripts - configMap: defaultMode: 493 name: sidecar-vanilla-postgresql-reload-tools-script-vanilla-kzgbtp name: cm-script-vanilla-postgresql-configuration - configMap: defaultMode: 493 name: sidecar-vanilla-kzgbtp-postgresql-config-manager-config name: config-manager-config - emptyDir: *** name: kb-tools - name: data persistentVolumeClaim: claimName: data-vanilla-kzgbtp-postgresql-1 - emptyDir: *** name: kubeblocks - name: kube-api-access-9s4mc projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2025-06-19T12:10:23Z" status: "True" type: PodReadyToStartContainers - lastProbeTime: null lastTransitionTime: "2025-06-19T12:10:25Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2025-06-19T12:31:24Z" message: 'containers with unready status: [postgresql]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2025-06-19T12:31:24Z" message: 'containers with unready status: [postgresql]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2025-06-19T12:10:22Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://0fc5f7228e0828d9d95e13f2199eeda4815a4c4a502a8e7cacfd2e497f7e0092 image: docker.io/apecloud/postgres:14.7 imageID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c lastState: *** name: config-manager ready: true restartCount: 0 started: true state: running: startedAt: "2025-06-19T12:10:25Z" volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true recursiveReadOnly: Disabled - containerID: containerd://b4e8391dd4b8207b0ad372a53bf84296168b2ae97800149eeb1a50caab224a6e image: docker.io/apecloud/kubeblocks-tools:0.9.4 imageID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 lastState: *** name: lorry ready: true restartCount: 0 started: true state: running: startedAt: "2025-06-19T12:10:25Z" volumeMounts: - mountPath: /postgresql/volume_data name: data - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true recursiveReadOnly: Disabled - containerID: containerd://70a9d6985fe891a382a028e18660376096bf7752e2b61e4eb716e68f00d38e6a image: docker.io/apecloud/postgres:14.7 imageID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c lastState: terminated: containerID: containerd://70a9d6985fe891a382a028e18660376096bf7752e2b61e4eb716e68f00d38e6a exitCode: 2 finishedAt: "2025-06-19T12:31:24Z" reason: Error startedAt: "2025-06-19T12:31:23Z" name: postgresql ready: false restartCount: 9 started: false state: waiting: message: back-off 5m0s restarting failed container=postgresql pod=vanilla-kzgbtp-postgresql-1_ns-auxmm(e11bcf51-ea66-478a-995a-2dc548b6c9e5) reason: CrashLoopBackOff volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /postgresql/volume_data name: data - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /kb-scripts name: scripts - mountPath: /kubeblocks name: kubeblocks - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true recursiveReadOnly: Disabled hostIP: 172.31.14.226 hostIPs: - ip: 172.31.14.226 initContainerStatuses: - containerID: containerd://02390d95c564f00f3e5b1f8fb12589cd7759eb6920134bb92b5a63d8257c5a03 image: docker.io/apecloud/syncer:0.3.6 imageID: docker.io/apecloud/syncer@sha256:d8cb38ff8f9e025e6211725bb61ac1cbaadb10dc5e9dc6a2613ef09f7a42f26e lastState: *** name: init-syncer ready: true restartCount: 0 started: false state: terminated: containerID: containerd://02390d95c564f00f3e5b1f8fb12589cd7759eb6920134bb92b5a63d8257c5a03 exitCode: 0 finishedAt: "2025-06-19T12:10:23Z" reason: Completed startedAt: "2025-06-19T12:10:23Z" volumeMounts: - mountPath: /kubeblocks name: kubeblocks - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true recursiveReadOnly: Disabled - containerID: containerd://12a29b6cfa04cf9cf84e2561c9e9ce407bb358ca0c28af8ec5b876654e7ba2a6 image: docker.io/apecloud/kubeblocks-tools:0.9.4 imageID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 lastState: *** name: install-config-manager-tool ready: true restartCount: 0 started: false state: terminated: containerID: containerd://12a29b6cfa04cf9cf84e2561c9e9ce407bb358ca0c28af8ec5b876654e7ba2a6 exitCode: 0 finishedAt: "2025-06-19T12:10:24Z" reason: Completed startedAt: "2025-06-19T12:10:24Z" volumeMounts: - mountPath: /postgresql/mount_conf name: postgresql-config - mountPath: /opt/kb-tools/reload/vanilla-postgresql-configuration name: cm-script-vanilla-postgresql-configuration - mountPath: /opt/config-manager name: config-manager-config - mountPath: /kb_tools name: kb-tools - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-9s4mc readOnly: true recursiveReadOnly: Disabled phase: Running podIP: 172.31.2.90 podIPs: - ip: 172.31.2.90 qosClass: Burstable startTime: "2025-06-19T12:10:22Z" ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe pod vanilla-kzgbtp-postgresql-0 vanilla-kzgbtp-postgresql-1-------------------------------------- `kubectl describe pod vanilla-kzgbtp-postgresql-0 --namespace ns-auxmm ` Name: vanilla-kzgbtp-postgresql-0 Namespace: ns-auxmm Priority: 0 Service Account: kb-vanilla-kzgbtp Node: ip-172-31-6-29.us-west-2.compute.internal/172.31.6.29 Start Time: Thu, 19 Jun 2025 18:31:40 +0800 Labels: app.kubernetes.io/component=vanilla-postgresql-14 app.kubernetes.io/instance=vanilla-kzgbtp app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=vanilla-postgresql-14 app.kubernetes.io/version=vanilla-postgresql-14 apps.kubeblocks.io/cluster-uid=9fe00cca-f01b-4f4e-8e1c-f33f8591d154 apps.kubeblocks.io/component-name=postgresql apps.kubeblocks.io/pod-name=vanilla-kzgbtp-postgresql-0 clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= componentdefinition.kubeblocks.io/name=vanilla-postgresql-14 controller-revision-hash=6c8d9d7bf5 kubeblocks.io/role=primary workloads.kubeblocks.io/access-mode=ReadWrite workloads.kubeblocks.io/instance=vanilla-kzgbtp-postgresql workloads.kubeblocks.io/managed-by=InstanceSet Annotations: apps.kubeblocks.io/component-replicas: 2 apps.kubeblocks.io/last-role-snapshot-version: 1750329814096971 Status: Running IP: 172.31.11.240 IPs: IP: 172.31.11.240 Controlled By: InstanceSet/vanilla-kzgbtp-postgresql Init Containers: init-syncer: Container ID: containerd://5e5278ee7a8be5bdf9eef18a2c7b8cc17cdece0282eb90c489143095bc5c5e9f Image: docker.io/apecloud/syncer:0.3.6 Image ID: docker.io/apecloud/syncer@sha256:d8cb38ff8f9e025e6211725bb61ac1cbaadb10dc5e9dc6a2613ef09f7a42f26e Port: Host Port: Command: sh -c cp -r /bin/syncer /kubeblocks/ State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 19 Jun 2025 18:31:44 +0800 Finished: Thu, 19 Jun 2025 18:31:44 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc Mounts: /kubeblocks from kubeblocks (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) install-config-manager-tool: Container ID: containerd://4d462f4a83b1519c07e61de32cee6a50957374b57f9975eceeec7723ab1fa807 Image: docker.io/apecloud/kubeblocks-tools:0.9.4 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 Port: Host Port: Command: cp /bin/reloader /kb_tools State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 19 Jun 2025 18:31:45 +0800 Finished: Thu, 19 Jun 2025 18:31:45 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc Mounts: /kb_tools from kb-tools (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/vanilla-postgresql-configuration from cm-script-vanilla-postgresql-configuration (rw) /postgresql/mount_conf from postgresql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) Containers: postgresql: Container ID: containerd://fe46f97f33678fa6ae8cf97d6d4e3035f3a031c1935e8e3a6e427d99071bf81e Image: docker.io/apecloud/postgres:14.7 Image ID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c Port: 5432/TCP Host Port: 0/TCP Command: /kubeblocks/syncer --port 3601 -- docker-entrypoint.sh --config-file=/postgresql/volume_data/conf/postgresql.conf --hba_file=/postgresql/volume_data/conf/pg_hba.conf State: Running Started: Thu, 19 Jun 2025 18:41:37 +0800 Last State: Terminated Reason: Error Exit Code: 2 Started: Thu, 19 Jun 2025 18:41:15 +0800 Finished: Thu, 19 Jun 2025 18:41:15 +0800 Ready: True Restart Count: 7 Limits: cpu: 600m memory: 644245094400m Requests: cpu: 600m memory: 644245094400m Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc ALLOW_NOSSL: true POSTGRESQL_PORT_NUMBER: 5432 PGDATA: /postgresql/volume_data/data PGCONF: /postgresql/volume_data/conf POSTGRESQL_MOUNTED_CONF_DIR: /postgresql/mount_conf PGUSER: $(POSTGRES_USER) PGPASSWORD: $(POSTGRES_PASSWORD) KB_ENGINE_TYPE: vanilla-postgresql Mounts: /dev/shm from dshm (rw) /kb-scripts from scripts (rw) /kb_tools from kb-tools (rw) /kubeblocks from kubeblocks (rw) /postgresql/mount_conf from postgresql-config (rw) /postgresql/volume_data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) lorry: Container ID: containerd://fa38d6cf08bb686721e3f36bc4e802c5b83c45ea15d8c4b62134e4f1153a1a6d Image: docker.io/apecloud/kubeblocks-tools:0.9.4 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: lorry --port 3501 --grpcport 50001 State: Running Started: Thu, 19 Jun 2025 18:41:37 +0800 Last State: Terminated Reason: StartError Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "lorry": executable file not found in $PATH: unknown Exit Code: 128 Started: Thu, 01 Jan 1970 08:00:00 +0800 Finished: Thu, 19 Jun 2025 18:41:15 +0800 Ready: True Restart Count: 6 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Readiness: http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc KB_RSM_ROLE_PROBE_PERIOD: 1 KB_BUILTIN_HANDLER: postgresql KB_SERVICE_USER: Optional: false KB_SERVICE_PASSWORD: Optional: false KB_SERVICE_PORT: 5432 KB_DATA_PATH: /postgresql/volume_data KB_RSM_ACTION_SVC_LIST: null KB_RSM_ROLE_UPDATE_MECHANISM: DirectAPIServerEventUpdate KB_RSM_ROLE_PROBE_TIMEOUT: 1 KB_CLUSTER_NAME: (v1:metadata.labels['app.kubernetes.io/instance']) KB_COMP_NAME: (v1:metadata.labels['apps.kubeblocks.io/component-name']) KB_SERVICE_CHARACTER_TYPE: postgresql Mounts: /postgresql/volume_data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) config-manager: Container ID: containerd://4a559d5430038eadb6ec4f2b6654901ec74628489218239605b3f1d1ae551206 Image: docker.io/apecloud/postgres:14.7 Image ID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c Port: 9901/TCP Host Port: 0/TCP Command: env Args: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) /kb_tools/reloader --log-level info --operator-update-enable --tcp 9901 --config /opt/config-manager/config-manager.yaml State: Running Started: Thu, 19 Jun 2025 18:41:37 +0800 Last State: Terminated Reason: StartError Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "env": executable file not found in $PATH: unknown Exit Code: 128 Started: Thu, 01 Jan 1970 08:00:00 +0800 Finished: Thu, 19 Jun 2025 18:41:16 +0800 Ready: True Restart Count: 6 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc CONFIG_MANAGER_POD_IP: (v1:status.podIP) TOOLS_PATH: /opt/kb-tools/reload/vanilla-postgresql-configuration:/opt/config-manager:/kb_tools Mounts: /kb_tools from kb-tools (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/vanilla-postgresql-configuration from cm-script-vanilla-postgresql-configuration (rw) /postgresql/mount_conf from postgresql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lfdrn (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: dshm: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: 644245094400m postgresql-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: vanilla-kzgbtp-postgresql-vanilla-postgresql-configuration Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: vanilla-kzgbtp-postgresql-vanilla-postgresql-scripts Optional: false cm-script-vanilla-postgresql-configuration: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-vanilla-postgresql-reload-tools-script-vanilla-kzgbtp Optional: false config-manager-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-vanilla-kzgbtp-postgresql-config-manager-config Optional: false kb-tools: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-vanilla-kzgbtp-postgresql-0 ReadOnly: false kubeblocks: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-lfdrn: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kb-data=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ------------------------------------------------------------------------------------------------------------------ `kubectl describe pod vanilla-kzgbtp-postgresql-1 --namespace ns-auxmm ` Name: vanilla-kzgbtp-postgresql-1 Namespace: ns-auxmm Priority: 0 Service Account: kb-vanilla-kzgbtp Node: ip-172-31-14-226.us-west-2.compute.internal/172.31.14.226 Start Time: Thu, 19 Jun 2025 20:10:22 +0800 Labels: app.kubernetes.io/component=vanilla-postgresql-14 app.kubernetes.io/instance=vanilla-kzgbtp app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=vanilla-postgresql-14 app.kubernetes.io/version=vanilla-postgresql-14 apps.kubeblocks.io/cluster-uid=9fe00cca-f01b-4f4e-8e1c-f33f8591d154 apps.kubeblocks.io/component-name=postgresql apps.kubeblocks.io/pod-name=vanilla-kzgbtp-postgresql-1 clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= componentdefinition.kubeblocks.io/name=vanilla-postgresql-14 controller-revision-hash=6bfcc759f workloads.kubeblocks.io/instance=vanilla-kzgbtp-postgresql workloads.kubeblocks.io/managed-by=InstanceSet Annotations: apps.kubeblocks.io/component-replicas: 2 kubeblocks.io/restart: 2025-06-19T11:16:07Z Status: Running IP: 172.31.2.90 IPs: IP: 172.31.2.90 Controlled By: InstanceSet/vanilla-kzgbtp-postgresql Init Containers: init-syncer: Container ID: containerd://02390d95c564f00f3e5b1f8fb12589cd7759eb6920134bb92b5a63d8257c5a03 Image: docker.io/apecloud/syncer:0.3.6 Image ID: docker.io/apecloud/syncer@sha256:d8cb38ff8f9e025e6211725bb61ac1cbaadb10dc5e9dc6a2613ef09f7a42f26e Port: Host Port: Command: sh -c cp -r /bin/syncer /kubeblocks/ State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 19 Jun 2025 20:10:23 +0800 Finished: Thu, 19 Jun 2025 20:10:23 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc Mounts: /kubeblocks from kubeblocks (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9s4mc (ro) install-config-manager-tool: Container ID: containerd://12a29b6cfa04cf9cf84e2561c9e9ce407bb358ca0c28af8ec5b876654e7ba2a6 Image: docker.io/apecloud/kubeblocks-tools:0.9.4 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 Port: Host Port: Command: cp /bin/reloader /kb_tools State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 19 Jun 2025 20:10:24 +0800 Finished: Thu, 19 Jun 2025 20:10:24 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc Mounts: /kb_tools from kb-tools (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/vanilla-postgresql-configuration from cm-script-vanilla-postgresql-configuration (rw) /postgresql/mount_conf from postgresql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9s4mc (ro) Containers: postgresql: Container ID: containerd://70a9d6985fe891a382a028e18660376096bf7752e2b61e4eb716e68f00d38e6a Image: docker.io/apecloud/postgres:14.7 Image ID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c Port: 5432/TCP Host Port: 0/TCP Command: /kubeblocks/syncer --port 3601 -- docker-entrypoint.sh --config-file=/postgresql/volume_data/conf/postgresql.conf --hba_file=/postgresql/volume_data/conf/pg_hba.conf State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Thu, 19 Jun 2025 20:31:23 +0800 Finished: Thu, 19 Jun 2025 20:31:24 +0800 Ready: False Restart Count: 9 Limits: cpu: 600m memory: 644245094400m Requests: cpu: 600m memory: 644245094400m Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc ALLOW_NOSSL: true POSTGRESQL_PORT_NUMBER: 5432 PGDATA: /postgresql/volume_data/data PGCONF: /postgresql/volume_data/conf POSTGRESQL_MOUNTED_CONF_DIR: /postgresql/mount_conf PGUSER: $(POSTGRES_USER) PGPASSWORD: $(POSTGRES_PASSWORD) KB_ENGINE_TYPE: vanilla-postgresql Mounts: /dev/shm from dshm (rw) /kb-scripts from scripts (rw) /kb_tools from kb-tools (rw) /kubeblocks from kubeblocks (rw) /postgresql/mount_conf from postgresql-config (rw) /postgresql/volume_data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9s4mc (ro) lorry: Container ID: containerd://b4e8391dd4b8207b0ad372a53bf84296168b2ae97800149eeb1a50caab224a6e Image: docker.io/apecloud/kubeblocks-tools:0.9.4 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:68643125a4ffc56bfe4463bf67908f51a236617ea10f0dfb9c2d4500022512c4 Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: lorry --port 3501 --grpcport 50001 State: Running Started: Thu, 19 Jun 2025 20:10:25 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Readiness: http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc KB_RSM_ROLE_PROBE_PERIOD: 1 KB_BUILTIN_HANDLER: postgresql KB_SERVICE_USER: Optional: false KB_SERVICE_PASSWORD: Optional: false KB_SERVICE_PORT: 5432 KB_DATA_PATH: /postgresql/volume_data KB_RSM_ACTION_SVC_LIST: null KB_RSM_ROLE_UPDATE_MECHANISM: DirectAPIServerEventUpdate KB_RSM_ROLE_PROBE_TIMEOUT: 1 KB_CLUSTER_NAME: (v1:metadata.labels['app.kubernetes.io/instance']) KB_COMP_NAME: (v1:metadata.labels['apps.kubeblocks.io/component-name']) KB_SERVICE_CHARACTER_TYPE: postgresql Mounts: /postgresql/volume_data from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9s4mc (ro) config-manager: Container ID: containerd://0fc5f7228e0828d9d95e13f2199eeda4815a4c4a502a8e7cacfd2e497f7e0092 Image: docker.io/apecloud/postgres:14.7 Image ID: docker.io/apecloud/postgres@sha256:5ac16ee311340b09e3670d660c76f77a611202fd07b05d486e934eece99bea7c Port: 9901/TCP Host Port: 0/TCP Command: env Args: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) /kb_tools/reloader --log-level info --operator-update-enable --tcp 9901 --config /opt/config-manager/config-manager.yaml State: Running Started: Thu, 19 Jun 2025 20:10:25 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: vanilla-kzgbtp-postgresql-env ConfigMap Optional: false vanilla-kzgbtp-postgresql-rsm-env ConfigMap Optional: false Environment: POSTGRES_USER: Optional: false POSTGRES_PASSWORD: Optional: false KB_POD_NAME: vanilla-kzgbtp-postgresql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: ns-auxmm (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).vanilla-kzgbtp-postgresql-headless.$(KB_NAMESPACE).svc CONFIG_MANAGER_POD_IP: (v1:status.podIP) TOOLS_PATH: /opt/kb-tools/reload/vanilla-postgresql-configuration:/opt/config-manager:/kb_tools Mounts: /kb_tools from kb-tools (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/vanilla-postgresql-configuration from cm-script-vanilla-postgresql-configuration (rw) /postgresql/mount_conf from postgresql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9s4mc (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: dshm: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: 644245094400m postgresql-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: vanilla-kzgbtp-postgresql-vanilla-postgresql-configuration Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: vanilla-kzgbtp-postgresql-vanilla-postgresql-scripts Optional: false cm-script-vanilla-postgresql-configuration: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-vanilla-postgresql-reload-tools-script-vanilla-kzgbtp Optional: false config-manager-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-vanilla-kzgbtp-postgresql-config-manager-config Optional: false kb-tools: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-vanilla-kzgbtp-postgresql-1 ReadOnly: false kubeblocks: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-9s4mc: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kb-data=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 24m default-scheduler Successfully assigned ns-auxmm/vanilla-kzgbtp-postgresql-1 to ip-172-31-14-226.us-west-2.compute.internal Normal Pulled 24m kubelet Container image "docker.io/apecloud/syncer:0.3.6" already present on machine Normal Created 24m kubelet Created container: init-syncer Normal Started 24m kubelet Started container init-syncer Normal Pulled 24m kubelet Container image "docker.io/apecloud/kubeblocks-tools:0.9.4" already present on machine Normal Created 24m kubelet Created container: install-config-manager-tool Normal Started 24m kubelet Started container install-config-manager-tool Normal Started 24m kubelet Started container lorry Normal Pulled 24m kubelet Container image "docker.io/apecloud/kubeblocks-tools:0.9.4" already present on machine Normal Created 24m kubelet Created container: lorry Normal Pulled 24m kubelet Container image "docker.io/apecloud/postgres:14.7" already present on machine Normal Created 24m kubelet Created container: config-manager Normal Started 24m kubelet Started container config-manager Normal Created 23m (x4 over 24m) kubelet Created container: postgresql Normal Started 23m (x4 over 24m) kubelet Started container postgresql Warning BackOff 3m53s (x97 over 24m) kubelet Back-off restarting failed container postgresql in pod vanilla-kzgbtp-postgresql-1_ns-auxmm(e11bcf51-ea66-478a-995a-2dc548b6c9e5) Normal Pulled 3m11s (x10 over 24m) kubelet Container image "docker.io/apecloud/postgres:14.7" already present on machine ------------------------------------------------------------------------------------------------------------------ --------------------------------------pod vanilla-kzgbtp-postgresql-0 vanilla-kzgbtp-postgresql-1-------------------------------------- `kubectl logs vanilla-kzgbtp-postgresql-0 --namespace ns-auxmm --tail 500` Defaulted container "postgresql" out of: postgresql, lorry, config-manager, init-syncer (init), install-config-manager-tool (init) == DB ERR == 2025-06-19 12:32:32.096 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:33.097 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:34.097 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:35.098 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:36.098 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:37.099 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:38.100 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:39.101 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:40.101 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:41.102 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:42.126 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:43.102 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:44.103 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:45.104 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == 2025-06-19T12:32:45Z INFO HA This member is Cluster's leader 2025-06-19T12:32:45Z DEBUG HA Refresh leader ttl == DB ERR == 2025-06-19 12:32:46.104 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:47.104 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:48.105 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:49.105 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:50.105 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:51.105 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:52.106 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:53.268 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:54.107 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:55.107 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:56.108 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:57.108 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:58.109 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:32:59.109 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:00.109 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:01.109 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:02.110 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:03.111 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:04.135 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:05.111 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:06.112 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:07.113 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:08.114 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:09.115 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:10.115 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:11.116 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:12.117 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:13.117 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:14.118 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:15.141 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:16.119 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:17.119 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:18.119 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:19.119 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:20.120 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:21.121 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:22.122 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:23.122 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:24.122 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:25.122 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:26.156 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:27.123 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:28.123 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:29.123 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:30.123 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:31.124 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:32.124 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:33.125 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:34.125 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:35.125 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:36.125 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:37.151 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:38.127 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:39.127 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:40.129 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:41.128 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:42.128 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:43.128 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:44.129 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:45.130 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == 2025-06-19T12:33:45Z INFO HA This member is Cluster's leader 2025-06-19T12:33:45Z DEBUG HA Refresh leader ttl == DB ERR == 2025-06-19 12:33:46.129 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:47.130 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:48.163 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:49.131 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:50.132 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:51.133 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:52.134 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:53.134 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:54.135 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:55.136 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:56.137 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:57.137 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:58.137 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:33:59.160 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:00.138 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:01.139 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:02.139 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:03.141 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:04.142 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:05.142 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:06.142 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:07.143 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:08.145 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:09.145 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:10.171 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:11.146 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:12.148 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:13.147 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:14.147 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:15.148 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:16.149 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:17.149 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:18.149 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:19.150 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:20.151 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:21.179 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:22.155 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:23.153 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:24.154 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:25.155 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:26.157 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:27.156 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:28.157 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:29.157 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:30.159 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:31.160 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:32.183 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:33.160 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:34.161 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:35.161 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == ------------------------------------------------------------------------------------------------------------------ `kubectl logs vanilla-kzgbtp-postgresql-1 --namespace ns-auxmm --tail 500` Defaulted container "postgresql" out of: postgresql, lorry, config-manager, init-syncer (init), install-config-manager-tool (init) 2025-06-19T12:31:23Z DEBUG Starting syncer ***"version": "4e24a0c83c87ff35927d5730f6616bc6f00e2f88"*** 2025-06-19T12:31:23Z INFO Initialize DB manager 2025-06-19T12:31:23Z DEBUG KB_WORKLOAD_TYPE ENV not set 2025-06-19T12:31:23Z INFO HTTPServer Starting HTTP Server 2025-06-19T12:31:23Z INFO HTTPServer API route path ***"method": "GET", "path": ["/v1.0/getrole", "/v1.0/datasync"]*** 2025-06-19T12:31:23Z INFO HTTPServer API route path ***"method": "POST", "path": ["/v1.0/rebuild", "/v1.0/start", "/v1.0/stop", "/v1.0/switchover"]*** 2025-06-19T12:31:23Z INFO HTTPServer http server ***"listen address": "0.0.0.0", "port": 3601*** 2025-06-19T12:31:23Z INFO HA HA starting 2025-06-19T12:31:23Z INFO pinger Waiting for dns resolution to be ready 2025-06-19T12:31:23Z INFO pinger dns resolution is ready ***"dns": "vanilla-kzgbtp-postgresql-1.vanilla-kzgbtp-postgresql-headless.ns-auxmm.svc"*** 2025-06-19T12:31:23Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=vanilla-kzgbtp,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=postgresql 2025-06-19T12:31:23Z INFO DCS-K8S podlist: 2 2025-06-19T12:31:23Z DEBUG HA cluster info ***"cluster": ***"ClusterCompName":"vanilla-kzgbtp-postgresql","Namespace":"ns-auxmm","Replicas":2,"HaConfig":***"DeleteMembers":***,"Leader":***"DBState":***"OpTimestamp":168769672,"Extra":***"replication_mode":"off","timeline":"10"***,"Index":"119894","Name":"vanilla-kzgbtp-postgresql-0","AcquireTime":1750331751,"RenewTime":1750336283,"TTL":15,"Resource":***"metadata":***"name":"vanilla-kzgbtp-postgresql-leader","namespace":"ns-auxmm","uid":"f9f4e470-b0b3-439f-bbee-fa6df200607b","resourceVersion":"119894","creationTimestamp":"2025-06-19T10:16:02Z","labels":***"app.kubernetes.io/instance":"vanilla-kzgbtp","app.kubernetes.io/managed-by":"kubeblocks","apps.kubeblocks.io/component-name":"postgresql"***,"annotations":***"acquire-time":"1750331751","dbstate":"***\"OpTimestamp\":168769672,\"Extra\":***\"replication_mode\":\"off\",\"timeline\":\"10\"***","leader":"vanilla-kzgbtp-postgresql-0","renew-time":"1750336283","ttl":"15"***,"ownerReferences":[***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","name":"vanilla-kzgbtp","uid":"9fe00cca-f01b-4f4e-8e1c-f33f8591d154"***],"managedFields":[***"manager":"syncer","operation":"Update","apiVersion":"v1","time":"2025-06-19T12:31:23Z","fieldsType":"FieldsV1","fieldsV1":***"f:metadata":***"f:annotations":***".":***,"f:acquire-time":***,"f:dbstate":***,"f:leader":***,"f:renew-time":***,"f:ttl":***,"f:labels":***".":***,"f:app.kubernetes.io/instance":***,"f:app.kubernetes.io/managed-by":***,"f:apps.kubeblocks.io/component-name":***,"f:ownerReferences":***".":***,"k:***\"uid\":\"9fe00cca-f01b-4f4e-8e1c-f33f8591d154\"***":***]***,"Members":[***"Index":"","Name":"vanilla-kzgbtp-postgresql-0","Role":"primary","PodIP":"172.31.11.240","DBPort":"5432","SyncerPort":"3601","UID":"f3b3a75d-ecb7-4978-baeb-a7597f9d8a14","ComponentName":"postgresql","UseIP":false***,***"Index":"","Name":"vanilla-kzgbtp-postgresql-1","Role":"","PodIP":"172.31.2.90","DBPort":"5432","SyncerPort":"3601","UID":"e11bcf51-ea66-478a-995a-2dc548b6c9e5","ComponentName":"postgresql","UseIP":false***],"Switchover":null,"Extra":null,"Resource":***"metadata":***"name":"vanilla-kzgbtp","namespace":"ns-auxmm","uid":"9fe00cca-f01b-4f4e-8e1c-f33f8591d154","resourceVersion":"119511","generation":10,"creationTimestamp":"2025-06-19T10:15:26Z","labels":***"app.kubernetes.io/instance":"vanilla-kzgbtp","clusterdefinition.kubeblocks.io/name":"vanilla-postgresql","clusterversion.kubeblocks.io/name":""***,"annotations":***"kubeblocks.io/reconcile":"2025-06-19T12:30:41.285628969Z","kubectl.kubernetes.io/last-applied-configuration":"***\"apiVersion\":\"apps.kubeblocks.io/v1alpha1\",\"kind\":\"Cluster\",\"metadata\":***\"annotations\":***,\"name\":\"vanilla-kzgbtp\",\"namespace\":\"ns-auxmm\"***,\"spec\":***\"clusterDefinitionRef\":\"vanilla-postgresql\",\"componentSpecs\":[***\"name\":\"postgresql\",\"replicas\":2,\"resources\":***\"limits\":***\"cpu\":\"500m\",\"memory\":\"0.5Gi\"***,\"requests\":***\"cpu\":\"500m\",\"memory\":\"0.5Gi\"***,\"serviceVersion\":\"14.7.0\",\"volumeClaimTemplates\":[***\"name\":\"data\",\"spec\":***\"accessModes\":[\"ReadWriteOnce\"],\"resources\":***\"requests\":***\"storage\":\"3Gi\"***,\"storageClassName\":null***]***],\"terminationPolicy\":\"Halt\",\"topology\":\"vanilla-postgresql\"***\n"***,"finalizers":["cluster.kubeblocks.io/finalizer"],"managedFields":[***"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2025-06-19T10:15:26Z","fieldsType":"FieldsV1","fieldsV1":***"f:metadata":***"f:annotations":***".":***,"f:kubectl.kubernetes.io/last-applied-configuration":***,"f:spec":***".":***,"f:clusterDefinitionRef":***,"f:topology":***,***"manager":"kbcli","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2025-06-19T12:16:15Z","fieldsType":"FieldsV1","fieldsV1":***"f:metadata":***"f:labels":***"f:app.kubernetes.io/instance":***,"f:spec":***"f:terminationPolicy":***,***"manager":"kubeblocks","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2025-06-19T12:26:31Z","fieldsType":"FieldsV1","fieldsV1":***"f:status":***".":***,"f:clusterDefGeneration":***,"f:components":***".":***,"f:postgresql":***".":***,"f:message":***".":***,"f:InstanceSet/vanilla-kzgbtp-postgresql":***,"f:phase":***,"f:podsReady":***,"f:podsReadyTime":***,"f:conditions":***,"f:observedGeneration":***,"f:phase":***,"subresource":"status"***,***"manager":"kubeblocks","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2025-06-19T12:30:41Z","fieldsType":"FieldsV1","fieldsV1":***"f:metadata":***"f:annotations":***"f:kubeblocks.io/reconcile":***,"f:finalizers":***".":***,"v:\"cluster.kubeblocks.io/finalizer\"":***,"f:labels":***".":***,"f:clusterdefinition.kubeblocks.io/name":***,"f:clusterversion.kubeblocks.io/name":***,"f:spec":***"f:componentSpecs":***,"f:resources":***".":***,"f:cpu":***,"f:memory":***,"f:services":***,"f:storage":***".":***,"f:size":***]***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","terminationPolicy":"WipeOut","componentSpecs":[***"name":"postgresql","componentDef":"vanilla-postgresql-14","replicas":2,"resources":***"limits":***"cpu":"600m","memory":"644245094400m"***,"requests":***"cpu":"600m","memory":"644245094400m"***,"volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"6Gi"***]***],"services":[***"name":"postgresql-vpc","serviceName":"postgresql-vpc","annotations":***"service.beta.kubernetes.io/aws-load-balancer-internal":"true","service.beta.kubernetes.io/aws-load-balancer-type":"nlb"***,"spec":***"ports":[***"name":"tcp-postgresql","protocol":"TCP","port":5432,"targetPort":"tcp-postgresql","nodePort":30267***],"type":"LoadBalancer"***,"roleSelector":"primary","componentSelector":"postgresql"***],"resources":***"cpu":"0","memory":"0"***,"storage":***"size":"0"***,"monitor":***,"status":***"observedGeneration":10,"phase":"Failed","components":***"postgresql":***"phase":"Failed","message":***"InstanceSet/vanilla-kzgbtp-postgresql":"[\"vanilla-kzgbtp-postgresql-1\"]"***,"podsReady":false,"podsReadyTime":"2025-06-19T11:15:52Z"***,"clusterDefGeneration":1,"conditions":[***"type":"ProvisioningStarted","status":"True","observedGeneration":10,"lastTransitionTime":"2025-06-19T10:15:26Z","reason":"PreCheckSucceed","message":"The operator has started the provisioning of Cluster: vanilla-kzgbtp"***,***"type":"ApplyResources","status":"True","observedGeneration":10,"lastTransitionTime":"2025-06-19T10:17:01Z","reason":"ApplyResourcesSucceed","message":"Successfully applied for resources"***,***"type":"ReplicasReady","status":"False","lastTransitionTime":"2025-06-19T11:16:07Z","reason":"ReplicasNotReady","message":"pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components"***,***"type":"Ready","status":"False","lastTransitionTime":"2025-06-19T11:16:07Z","reason":"ComponentsNotReady","message":"pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components"***]*** 2025-06-19T12:31:23Z INFO PostgreSQL get local timeline: 9, local lsn: 184549536, isRecovery: true 2025-06-19T12:31:24Z INFO PostgreSQL archive is not enabled panic: start DB service failed: exec command su failed, err:exit status 1, stderr:pg_rewind: fatal: restore_command is not set in the target cluster goroutine 52 [running]: github.com/apecloud/syncer/highavailability.(*Ha).Start(0xc000259dc0) /src/highavailability/ha.go:257 +0xa53 created by github.com/apecloud/syncer/highavailability.Start in goroutine 1 /src/highavailability/ha.go:69 +0x285 ------------------------------------------------------------------------------------------------------------------ cluster list-logs `kbcli cluster list-logs vanilla-kzgbtp --namespace ns-auxmm ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update vanilla-kzgbtp --enable-all-logs=true --namespace ns-auxmm Error from server (NotFound): pods "vanilla-kzgbtp-postgresql-0" not found cluster logs `kbcli cluster logs vanilla-kzgbtp --tail 30 --namespace ns-auxmm ` Defaulted container "postgresql" out of: postgresql, lorry, config-manager, init-syncer (init), install-config-manager-tool (init) == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:35.161 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:36.163 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:37.162 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:38.163 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:39.163 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:40.164 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-06-19 12:34:41.165 GMT [4804] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == Cluster check failed times reach 5, skip test lb. delete cluster vanilla-kzgbtp `kbcli cluster delete vanilla-kzgbtp --auto-approve --namespace ns-auxmm ` Cluster vanilla-kzgbtp deleted pod_info:vanilla-kzgbtp-postgresql-0 3/3 Terminating 19 (113m ago) 123m vanilla-kzgbtp-postgresql-1 2/3 Terminating 9 24m No resources found in ns-auxmm namespace. delete cluster pod done No resources found in ns-auxmm namespace. check cluster resource non-exist OK: pvc No resources found in ns-auxmm namespace. delete cluster done No resources found in ns-auxmm namespace. No resources found in ns-auxmm namespace. No resources found in ns-auxmm namespace. Vanilla Postgresql Test Suite All Done! --------------------------------------Vanilla Postgresql (Topology = vanilla-postgresql Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[Topology=vanilla-postgresql;ComponentVersion=vanilla-postgresql;ServiceVersion=14.7.0;]|[Description=Create a cluster with the specified topology vanilla-postgresql and component version vanilla-postgresql and service version 14.7.0] [PASSED]|[Connect]|[ComponentName=postgresql]|[Description=Connect to the cluster] [PASSED]|[Expose]|[Enable=true;TYPE=vpc;ComponentName=postgresql]|[Description=Expose Enable the vpc service with postgresql component] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=postgresql]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Bench]|[ComponentName=postgresql]|[Description=Bench the cluster service with postgresql component] [PASSED]|[Bench]|[HostType=LB;ComponentName=postgresql]|[Description=Bench the cluster LB service with postgresql component] [PASSED]|[VerticalScaling]|[ComponentName=postgresql]|[Description=VerticalScaling the cluster specify component postgresql] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=postgresql]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=postgresql]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=postgresql]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=postgresql]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=postgresql]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=postgresql]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [FAILED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=postgresql]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[VolumeExpansion]|[ComponentName=postgresql]|[Description=VolumeExpansion the cluster specify component postgresql] [FAILED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=postgresql]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [FAILED]|[Restart]|[-]|[Description=Restart the cluster] [FAILED]|[HorizontalScaling Out]|[ComponentName=postgresql]|[Description=HorizontalScaling Out the cluster specify component postgresql] [FAILED]|[HorizontalScaling In]|[ComponentName=postgresql]|[Description=HorizontalScaling In the cluster specify component postgresql] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=postgresql]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [FAILED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]