source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-dpmgs ` `kubectl create namespace ns-dpmgs` namespace/ns-dpmgs created create namespace ns-dpmgs done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 52.5M 0 --:--:-- --:--:-- --:--:-- 52.5M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:postgresql set component version No resources found no component version found set postgresql component definition set postgresql component definition postgresql-12 not found component version set replicas first:2 set replicas third:2 set minimum cmpv service version set minimum cmpv service version replicas:2 REPORT_COUNT:1 CLUSTER_TOPOLOGY:replication Not found topology in cluster definition postgresql LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 3 No resources found in ns-dpmgs namespace. termination_policy:Halt create 2 replica Halt postgresql cluster check cluster version check cluster definition check component definition set component definition by component version no component definitions found apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: postgres-jeiaru namespace: ns-dpmgs spec: terminationPolicy: Halt componentSpecs: - name: postgresql componentDef: postgresql-12 replicas: 2 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi updateStrategy: BestEffortParallel switchPolicy: type: Noop volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 3Gi `kubectl apply -f test_create_postgres-jeiaru.yaml` cluster.apps.kubeblocks.io/postgres-jeiaru created apply test_create_postgres-jeiaru.yaml Success `rm -rf test_create_postgres-jeiaru.yaml` check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Jun 19,2025 18:16 UTC+0800 cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=postgres-jeiaru` set secret: postgres-jeiaru-postgresql-account-postgres `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:451Sq78xLa;DB_PORT:5432;DB_DATABASE:postgres check pod postgres-jeiaru-postgresql-0 container_name postgresql exist password 451Sq78xLa check pod postgres-jeiaru-postgresql-0 container_name pgbouncer exist password 451Sq78xLa check pod postgres-jeiaru-postgresql-0 container_name exporter exist password 451Sq78xLa check pod postgres-jeiaru-postgresql-0 container_name lorry exist password 451Sq78xLa check pod postgres-jeiaru-postgresql-0 container_name config-manager exist password 451Sq78xLa No container logs contain secret password. describe cluster `kbcli cluster describe postgres-jeiaru --namespace ns-dpmgs ` Name: postgres-jeiaru Created Time: Jun 19,2025 18:16 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-dpmgs Running Halt Endpoints: COMPONENT MODE INTERNAL EXTERNAL postgresql ReadWrite postgres-jeiaru-postgresql-postgresql.ns-dpmgs.svc.cluster.local:5432 postgres-jeiaru-postgresql-postgresql.ns-dpmgs.svc.cluster.local:6432 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME postgresql postgres-jeiaru-postgresql-0 primary Running us-west-2a ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 postgresql postgres-jeiaru-postgresql-1 secondary Running us-west-2a ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:16 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS postgresql false 100m / 100m 512Mi / 512Mi data:3Gi kb-default-sc Images: COMPONENT TYPE IMAGE postgresql docker.io/apecloud/spilo:12.15.0-pgvector-v0.6.1 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-dpmgs postgres-jeiaru `kbcli cluster label postgres-jeiaru app.kubernetes.io/instance- --namespace ns-dpmgs ` label "app.kubernetes.io/instance" not found. `kbcli cluster label postgres-jeiaru app.kubernetes.io/instance=postgres-jeiaru --namespace ns-dpmgs ` `kbcli cluster label postgres-jeiaru --list --namespace ns-dpmgs ` NAME NAMESPACE LABELS postgres-jeiaru ns-dpmgs app.kubernetes.io/instance=postgres-jeiaru label cluster app.kubernetes.io/instance=postgres-jeiaru Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=postgres-jeiaru --namespace ns-dpmgs ` `kbcli cluster label postgres-jeiaru --list --namespace ns-dpmgs ` NAME NAMESPACE LABELS postgres-jeiaru ns-dpmgs app.kubernetes.io/instance=postgres-jeiaru case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label postgres-jeiaru case.name=kbcli.test2 --overwrite --namespace ns-dpmgs ` `kbcli cluster label postgres-jeiaru --list --namespace ns-dpmgs ` NAME NAMESPACE LABELS postgres-jeiaru ns-dpmgs app.kubernetes.io/instance=postgres-jeiaru case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label postgres-jeiaru case.name- --namespace ns-dpmgs ` `kbcli cluster label postgres-jeiaru --list --namespace ns-dpmgs ` NAME NAMESPACE LABELS postgres-jeiaru ns-dpmgs app.kubernetes.io/instance=postgres-jeiaru delete cluster label case.name Success cluster connect `echo 'create extension vector;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs --namespace ns-dpmgs ` Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file CREATE EXTENSION Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file List of installed extensions Name | Version | Schema | Description --------------------+---------+------------+----------------------------------------------------------- file_fdw | 1.0 | public | foreign-data wrapper for flat file access pg_auth_mon | 1.1 | public | monitor connection attempts per user pg_cron | 1.5 | pg_catalog | Job scheduler for PostgreSQL pg_stat_kcache | 2.2.3 | public | Kernel statistics gathering pg_stat_statements | 1.7 | public | track execution statistics of all SQL statements executed plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language plpython3u | 1.0 | pg_catalog | PL/Python3U untrusted procedural language set_user | 3.0 | public | similar to SET ROLE but with added logging vector | 0.6.1 | public | vector data type and ivfflat and hnsw access methods (9 rows) `echo 'show max_connections;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) max_connections ----------------- 56 (1 row) connect cluster Success set max_connections to 56 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-postgres-jeiaru" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-postgres-jeiaru --namespace ns-dpmgs ` Error from server (NotFound): pods "test-db-client-executionloop-postgres-jeiaru" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-postgres-jeiaru" not found `kubectl get secrets -l app.kubernetes.io/instance=postgres-jeiaru` set secret: postgres-jeiaru-postgresql-account-postgres `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:451Sq78xLa;DB_PORT:5432;DB_DATABASE:postgres apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-postgres-jeiaru namespace: ns-dpmgs spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "postgres-jeiaru-postgresql-postgresql.ns-dpmgs.svc.cluster.local" - "--user" - "postgres" - "--password" - "451Sq78xLa" - "--port" - "5432" - "--dbtype" - "postgresql" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-postgres-jeiaru.yaml` pod/test-db-client-executionloop-postgres-jeiaru created apply test-db-client-executionloop-postgres-jeiaru.yaml Success `rm -rf test-db-client-executionloop-postgres-jeiaru.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 0/1 ContainerCreating 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 16s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 39s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 45s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 57s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 63s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 68s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 1/1 Running 0 74s check pod test-db-client-executionloop-postgres-jeiaru status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-postgres-jeiaru 0/1 Completed 0 80s check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done --host postgres-jeiaru-postgresql-postgresql.ns-dpmgs.svc.cluster.local --user postgres --password 451Sq78xLa --port 5432 --dbtype postgresql --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:20:23.940 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.postgresql.jdbc.TimestampUtils (file:/app/oneclient-1.0-all.jar) to field java.util.TimeZone.defaultTimeZone WARNING: Please consider reporting this to the maintainers of org.postgresql.jdbc.TimestampUtils WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Execution loop start: create databases executions_loop CREATE DATABASE executions_loop; reconnect connection executions_loop drop table executions_loop_table DROP TABLE IF EXISTS executions_loop_table; create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop_table (id SERIAL PRIMARY KEY, value TEXT, tinyint_col SMALLINT, smallint_col SMALLINT, integer_col INTEGER, bigint_col BIGINT, real_col REAL, double_col DOUBLE PRECISION, numeric_col NUMERIC(10, 2), date_col DATE, time_col TIME, timestamp_col TIMESTAMP, timestamptz_col TIMESTAMP WITH TIME ZONE, interval_col INTERVAL, boolean_col BOOLEAN, char_col CHAR(10), varchar_col VARCHAR(255), text_col TEXT, bytea_col BYTEA, uuid_col UUID, json_col JSON, jsonb_col JSONB, xml_col XML, enum_col VARCHAR(10) CHECK (enum_col IN ('Option1', 'Option2', 'Option3')), set_col VARCHAR(255) CHECK (set_col IN ('Value1', 'Value2', 'Value3')), int_array_col INTEGER[], text_array_col TEXT[], point_col POINT, line_col LINE, lseg_col LSEG, box_col BOX, path_col PATH, polygon_col POLYGON, circle_col CIRCLE, cidr_col CIDR, inet_col INET, macaddr_col MACADDR, macaddr8_col MACADDR8, bit_col BIT(8), bit_var_col BIT VARYING(8), varbit_col BIT VARYING(8), money_col MONEY, oid_col OID, regproc_col REGPROC, regprocedure_col REGPROCEDURE, regoper_col REGOPER, regoperator_col REGOPERATOR, regclass_col REGCLASS, regtype_col REGTYPE, regrole_col REGROLE, regnamespace_col REGNAMESPACE, regconfig_col REGCONFIG, regdictionary_col REGDICTIONARY ); Execution loop start:INSERT INTO executions_loop_table (value, tinyint_col, smallint_col, integer_col, bigint_col, real_col, double_col, numeric_col, date_col, time_col, timestamp_col, timestamptz_col, interval_col, boolean_col, char_col, varchar_col, text_col, bytea_col, uuid_col, json_col, jsonb_col, xml_col, enum_col, set_col, int_array_col, text_array_col, point_col, line_col, lseg_col, box_col, path_col, polygon_col, circle_col, cidr_col, inet_col, macaddr_col, macaddr8_col, bit_col, bit_var_col, varbit_col, money_col, oid_col, regproc_col, regprocedure_col, regoper_col, regoperator_col, regclass_col, regtype_col, regrole_col, regnamespace_col, regconfig_col, regdictionary_col) VALUES ('executions_loop_test_1', 17, 6013, 1667385696, -3539596970858545883, 0.44092047, 0.4209686021298382, 20.213352400537367, '2025-06-19', '10:20:25', '2025-06-19 10:20:25.289', CURRENT_TIMESTAMP, '3 hours 48 minutes 32 seconds', FALSE, '2YDNDE5VJT', 'xddVHnKybMPE3wOg28DHy8kIxJfg3klDeRQk6PL5WHnK0GTHsCB3h38PVq1niPckWqk6BuBhFUjIrWnaEOMlUo3sX3n1O589x1lEjqLEHMqzbmP2BWkGZ3FDLwfdNKedcOwKNIAqSQJUaiWyj9k6t4UAu6qIxIHGkLRyGd2qveSZcwAqVCLGkBMeZanoiXpU3wgRjt18C9auqTJ6xlvfn995KSStANcpgmJ6J7ukdKXawnjkM6u0AU5tZ1H2oht', 'pzYOqiD0tLdCDXTPup4D0Pn3k98kGWEnzvwRAW9fTRQZA2F5acHtijHuCLZglRLCA9KQPaFh9BWyAID5mqPwvB4R6Rp7WFH3Y6KqaHHBti9p8dISM8nYGoxxN4E3FtR2sRcJKobMESPSruEQsBC7rYKvOIB9do0WJ66FwPB1AdvXZesDqiGEVLJeVM4onUzRdTtr1jZ6D4ZABFlIYwyp6Pu7BxbG5Ti6OKFvupvK10numUGbKGbpUsQZsODgGYP', decode('d48d8a5b85b3091f53cd', 'hex'), 'c4ca7ced-1ccf-44a1-ad6f-4880781ac3d3', '***"key1": "riSAeoTea3", "key2": 91***', '***"key1": "7ae1b0VHph", "key2": 20***', 'tTnnkBz5AA90', 'Option3', 'Value3', ARRAY[43, 42, 22], ARRAY['UFTmAfRmmV', 'M4M1DxzrmX', 'nrLzP91awY'], '(77.37064279446611, 14.068855548948378)', '***88.22258806319147, 81.16373411323909, 29.62475047265124***', '[(79.87131178107359, 46.45466218627031), (83.61836965201064, 73.53343026544152)]', '((93.76010046520528, 16.011731066365932), (74.33233850334318, 99.7852799215326))', '((62.1814475544526, 25.28756304463766), (74.41549906279728, 54.33527016195393), (48.15014491890989, 84.28258636170364))', '((55.260802037100035, 85.8127530649105), (57.31857963630647, 52.30974132699934), (18.41313946712385, 89.77288106612329), (11.795720684330313, 59.338512641758435))', '<(28.531835, 96.992476), 4.159161>', '192.168.54.0/24', '192.168.140.51', '08:00:2b:01:02:03', '08:00:2b:01:02:03:04:05', B'10101010', B'10101010', B'10101010', '$497.78397385131046', 222238494, 'acos', abs(1), '#-', +1, 'pg_class', 'integer', 'postgres', 'pg_catalog', 'simple', 'english_stem' ); [ 1s ] executions total: 1 successful: 1 failed: 0 disconnect: 0 [ 2s ] executions total: 70 successful: 70 failed: 0 disconnect: 0 [ 3s ] executions total: 134 successful: 134 failed: 0 disconnect: 0 [ 4s ] executions total: 201 successful: 201 failed: 0 disconnect: 0 [ 5s ] executions total: 280 successful: 280 failed: 0 disconnect: 0 [ 6s ] executions total: 349 successful: 349 failed: 0 disconnect: 0 [ 7s ] executions total: 407 successful: 407 failed: 0 disconnect: 0 [ 8s ] executions total: 419 successful: 419 failed: 0 disconnect: 0 [ 9s ] executions total: 495 successful: 495 failed: 0 disconnect: 0 [ 10s ] executions total: 554 successful: 554 failed: 0 disconnect: 0 [ 11s ] executions total: 615 successful: 615 failed: 0 disconnect: 0 [ 12s ] executions total: 662 successful: 662 failed: 0 disconnect: 0 [ 13s ] executions total: 726 successful: 726 failed: 0 disconnect: 0 [ 14s ] executions total: 790 successful: 790 failed: 0 disconnect: 0 [ 15s ] executions total: 856 successful: 856 failed: 0 disconnect: 0 [ 16s ] executions total: 928 successful: 928 failed: 0 disconnect: 0 [ 17s ] executions total: 983 successful: 983 failed: 0 disconnect: 0 [ 18s ] executions total: 1039 successful: 1039 failed: 0 disconnect: 0 [ 19s ] executions total: 1105 successful: 1105 failed: 0 disconnect: 0 [ 20s ] executions total: 1167 successful: 1167 failed: 0 disconnect: 0 [ 21s ] executions total: 1238 successful: 1238 failed: 0 disconnect: 0 [ 22s ] executions total: 1272 successful: 1272 failed: 0 disconnect: 0 [ 23s ] executions total: 1286 successful: 1286 failed: 0 disconnect: 0 [ 24s ] executions total: 1325 successful: 1325 failed: 0 disconnect: 0 [ 25s ] executions total: 1405 successful: 1405 failed: 0 disconnect: 0 [ 26s ] executions total: 1483 successful: 1483 failed: 0 disconnect: 0 [ 27s ] executions total: 1524 successful: 1524 failed: 0 disconnect: 0 [ 28s ] executions total: 1572 successful: 1572 failed: 0 disconnect: 0 [ 29s ] executions total: 1635 successful: 1635 failed: 0 disconnect: 0 [ 30s ] executions total: 1700 successful: 1700 failed: 0 disconnect: 0 [ 31s ] executions total: 1764 successful: 1764 failed: 0 disconnect: 0 [ 32s ] executions total: 1828 successful: 1828 failed: 0 disconnect: 0 [ 33s ] executions total: 1891 successful: 1891 failed: 0 disconnect: 0 [ 34s ] executions total: 1955 successful: 1955 failed: 0 disconnect: 0 [ 35s ] executions total: 2014 successful: 2014 failed: 0 disconnect: 0 [ 36s ] executions total: 2078 successful: 2078 failed: 0 disconnect: 0 [ 37s ] executions total: 2099 successful: 2099 failed: 0 disconnect: 0 [ 38s ] executions total: 2115 successful: 2115 failed: 0 disconnect: 0 [ 39s ] executions total: 2153 successful: 2153 failed: 0 disconnect: 0 [ 40s ] executions total: 2229 successful: 2229 failed: 0 disconnect: 0 [ 41s ] executions total: 2294 successful: 2294 failed: 0 disconnect: 0 [ 42s ] executions total: 2358 successful: 2358 failed: 0 disconnect: 0 [ 43s ] executions total: 2420 successful: 2420 failed: 0 disconnect: 0 [ 44s ] executions total: 2496 successful: 2496 failed: 0 disconnect: 0 [ 45s ] executions total: 2569 successful: 2569 failed: 0 disconnect: 0 [ 46s ] executions total: 2644 successful: 2644 failed: 0 disconnect: 0 [ 47s ] executions total: 2672 successful: 2672 failed: 0 disconnect: 0 [ 48s ] executions total: 2742 successful: 2742 failed: 0 disconnect: 0 [ 49s ] executions total: 2814 successful: 2814 failed: 0 disconnect: 0 [ 50s ] executions total: 2886 successful: 2886 failed: 0 disconnect: 0 [ 51s ] executions total: 2950 successful: 2950 failed: 0 disconnect: 0 [ 52s ] executions total: 2965 successful: 2965 failed: 0 disconnect: 0 [ 53s ] executions total: 2973 successful: 2973 failed: 0 disconnect: 0 [ 54s ] executions total: 2992 successful: 2992 failed: 0 disconnect: 0 [ 55s ] executions total: 3005 successful: 3005 failed: 0 disconnect: 0 [ 56s ] executions total: 3063 successful: 3063 failed: 0 disconnect: 0 [ 57s ] executions total: 3105 successful: 3105 failed: 0 disconnect: 0 [ 58s ] executions total: 3176 successful: 3176 failed: 0 disconnect: 0 [ 59s ] executions total: 3240 successful: 3240 failed: 0 disconnect: 0 [ 60s ] executions total: 3248 successful: 3248 failed: 0 disconnect: 0 Test Result: Total Executions: 3248 Successful Executions: 3248 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: postgresql Host: postgres-jeiaru-postgresql-postgresql.ns-dpmgs.svc.cluster.local Port: 5432 Database: Table: User: postgres Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 3248 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-postgres-jeiaru --namespace ns-dpmgs ` pod/test-db-client-executionloop-postgres-jeiaru patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-postgres-jeiaru" force deleted `echo "DROP TABLE IF EXISTS tmp_table; CREATE TABLE IF NOT EXISTS tmp_table (id INT PRIMARY KEY , value text); INSERT INTO tmp_table (id,value) VALUES (1,'lmpkt');" | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file NOTICE: table "tmp_table" does not exist, skipping DROP TABLE CREATE TABLE INSERT 0 1 add consistent data lmpkt Success `echo "DROP TABLE IF EXISTS tmp_table; CREATE TABLE IF NOT EXISTS tmp_table (id INT PRIMARY KEY , value text); INSERT INTO tmp_table (id,value) VALUES (1,'lmpkt');" | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file ERROR: cannot execute DROP TABLE in a read-only transaction ERROR: cannot execute CREATE TABLE in a read-only transaction ERROR: cannot execute INSERT in a read-only transaction check add consistent data readonly Success cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose postgres-jeiaru --auto-approve --force=true --type vpc --enable true --components postgresql --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-expose-sbccp created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-expose-sbccp -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-expose-sbccp ns-dpmgs Expose postgres-jeiaru postgresql Running 0/1 Jun 19,2025 18:21 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-expose-sbccp ns-dpmgs Expose postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 18:21 UTC+0800 check ops status done ops_status:postgres-jeiaru-expose-sbccp ns-dpmgs Expose postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 18:21 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-expose-sbccp --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-expose-sbccp patched `kbcli cluster delete-ops --name postgres-jeiaru-expose-sbccp --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-expose-sbccp deleted `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-1;secondary: postgres-jeiaru-postgresql-0 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check failover pod name failover pod name:postgres-jeiaru-postgresql-1 failover kill1 Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-0 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart postgres-jeiaru --auto-approve --force=true --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-restart-l427f created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-restart-l427f -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-restart-l427f ns-dpmgs Restart postgres-jeiaru postgresql Running 0/2 Jun 19,2025 18:23 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 18:24 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-7-10.us-west-2.compute.internal/172.31.7.10 Jun 19,2025 18:26 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-restart-l427f ns-dpmgs Restart postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:23 UTC+0800 check ops status done ops_status:postgres-jeiaru-restart-l427f ns-dpmgs Restart postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:23 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-restart-l427f --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-restart-l427f patched `kbcli cluster delete-ops --name postgres-jeiaru-restart-l427f --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-restart-l427f deleted `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cluster configure component_tmp: postgresql check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure postgres-jeiaru --auto-approve --force=true --set shared_buffers=512MB --components postgresql --config-spec postgresql-configuration --config-file postgresql.conf --namespace ns-dpmgs ` Will updated configure file meta: ConfigSpec: postgresql-configuration ConfigFile: postgresql.conf ComponentName: postgresql ClusterName: postgres-jeiaru OpsRequest postgres-jeiaru-reconfiguring-5wnrd created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-reconfiguring-5wnrd -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-reconfiguring-5wnrd ns-dpmgs Reconfiguring postgres-jeiaru postgresql Running -/- Jun 19,2025 18:28 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:30 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-7-10.us-west-2.compute.internal/172.31.7.10 Jun 19,2025 18:29 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-1;secondary: postgres-jeiaru-postgresql-0 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-reconfiguring-5wnrd ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:28 UTC+0800 check ops status done ops_status:postgres-jeiaru-reconfiguring-5wnrd ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:28 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-reconfiguring-5wnrd --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-reconfiguring-5wnrd patched `kbcli cluster delete-ops --name postgres-jeiaru-reconfiguring-5wnrd --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-reconfiguring-5wnrd deleted check config variables Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 512MB configure:[shared_buffers] result actual:[512MB] equal expected:[512MB] `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-0 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-7-10.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod postgres-jeiaru-postgresql-0 --namespace ns-dpmgs -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-1-34.us-west-2.compute.internal success kubectl drain ip-172-31-7-10.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-7-10.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-ll76c, kb-mqtcr/kb-addon-apecloud-otel-collector-rvrl9, kube-system/aws-node-dqkjr, kube-system/ebs-csi-node-5j62x, kube-system/kube-proxy-cswgv evicting pod ns-svryh/kingbase-jzvvix-backup-kingbase-2 evicting pod ns-dpmgs/postgres-jeiaru-postgresql-1 evicting pod ns-oooxt/kafka-pajpox-kafka-broker-2 pod/kingbase-jzvvix-backup-kingbase-2 evicted pod/postgres-jeiaru-postgresql-1 evicted pod/kafka-pajpox-kafka-broker-2 evicted node/ip-172-31-7-10.us-west-2.compute.internal drained kubectl uncordon ip-172-31-7-10.us-west-2.compute.internal node/ip-172-31-7-10.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:30 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:32 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover drainnode Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cluster update monitor KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster postgres-jeiaru --namespace ns-dpmgs --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/postgres-jeiaru patched check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:30 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:32 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done create curl-jq pod postgres-jeiaru-kb-curl `kubectl run --namespace kb-mqtcr postgres-jeiaru-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/postgres-jeiaru-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE postgres-jeiaru-kb-curl 0/1 ContainerCreating 0 1s check pod postgres-jeiaru-kb-curl status done `kubectl wait --for=condition=Ready pod postgres-jeiaru-kb-curl --namespace kb-mqtcr` pod/postgres-jeiaru-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE postgres-jeiaru-kb-curl 1/1 Running 0 6s kubectl exec -it postgres-jeiaru-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it postgres-jeiaru-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 122 100 122 0 0 37251 0 --:--:-- --:--:-- --:--:-- 40666 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750329302.190,"25.500000000000018"]***]*** k8s_pod_cpu_limit: 1750329302.190 monitor cluster Success `kubectl delete pod postgres-jeiaru-kb-curl --namespace kb-mqtcr --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "postgres-jeiaru-kb-curl" force deleted check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:30 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:32 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in postgres-jeiaru namespace. `kbcli cluster hscale postgres-jeiaru --auto-approve --force=true --components postgresql --replicas 3 --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-horizontalscaling-msz2q created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-horizontalscaling-msz2q -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-horizontalscaling-msz2q ns-dpmgs HorizontalScaling postgres-jeiaru postgresql Running 0/1 Jun 19,2025 18:35 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:30 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:32 UTC+0800 postgres-jeiaru-postgresql-2 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:35 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 postgres-jeiaru-postgresql-2 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done No resources found in postgres-jeiaru namespace. check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-horizontalscaling-msz2q ns-dpmgs HorizontalScaling postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 18:35 UTC+0800 check ops status done ops_status:postgres-jeiaru-horizontalscaling-msz2q ns-dpmgs HorizontalScaling postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 18:35 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-horizontalscaling-msz2q --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-horizontalscaling-msz2q patched `kbcli cluster delete-ops --name postgres-jeiaru-horizontalscaling-msz2q --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-horizontalscaling-msz2q deleted `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary), postgres-jeiaru-postgresql-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in postgres-jeiaru namespace. `kbcli cluster hscale postgres-jeiaru --auto-approve --force=true --components postgresql --replicas 2 --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-horizontalscaling-thkz9 created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-horizontalscaling-thkz9 -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-horizontalscaling-thkz9 ns-dpmgs HorizontalScaling postgres-jeiaru postgresql Running 0/1 Jun 19,2025 18:37 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:30 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:32 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done No resources found in postgres-jeiaru namespace. check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-horizontalscaling-thkz9 ns-dpmgs HorizontalScaling postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 18:37 UTC+0800 check ops status done ops_status:postgres-jeiaru-horizontalscaling-thkz9 ns-dpmgs HorizontalScaling postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 18:37 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-horizontalscaling-thkz9 --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-horizontalscaling-thkz9 patched `kbcli cluster delete-ops --name postgres-jeiaru-horizontalscaling-thkz9 --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-horizontalscaling-thkz9 deleted `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-postgres-jeiaru" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-postgres-jeiaru.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-postgres-jeiaru created apply test-chaos-mesh-timeoffset-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-timeoffset-postgres-jeiaru.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:30 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:32 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-postgres-jeiaru" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-postgres-jeiaru" not found check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover timeoffset Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cluster update enable-all-logs `kbcli cluster update postgres-jeiaru --enable-all-logs=true --namespace ns-dpmgs ` error: name is required check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:30 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:32 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop postgres-jeiaru --auto-approve --force=true --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-stop-cm2n6 created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-stop-cm2n6 -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-stop-cm2n6 ns-dpmgs Stop postgres-jeiaru postgresql Running 0/2 Jun 19,2025 18:41 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Stopping Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-stop-cm2n6 ns-dpmgs Stop postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:41 UTC+0800 check ops status done ops_status:postgres-jeiaru-stop-cm2n6 ns-dpmgs Stop postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:41 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-stop-cm2n6 --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-stop-cm2n6 patched `kbcli cluster delete-ops --name postgres-jeiaru-stop-cm2n6 --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-stop-cm2n6 deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start postgres-jeiaru --force=true --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-start-b7gr5 created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-start-b7gr5 -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-start-b7gr5 ns-dpmgs Start postgres-jeiaru postgresql Running 0/2 Jun 19,2025 18:42 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 18:42 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:42 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-start-b7gr5 ns-dpmgs Start postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:42 UTC+0800 check ops status done ops_status:postgres-jeiaru-start-b7gr5 ns-dpmgs Start postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:42 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-start-b7gr5 --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-start-b7gr5 patched `kbcli cluster delete-ops --name postgres-jeiaru-start-b7gr5 --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-start-b7gr5 deleted `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale postgres-jeiaru --auto-approve --force=true --components postgresql --cpu 200m --memory 0.6Gi --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-verticalscaling-6mwfp created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-verticalscaling-6mwfp -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-verticalscaling-6mwfp ns-dpmgs VerticalScaling postgres-jeiaru postgresql Running 0/2 Jun 19,2025 18:44 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:45 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-1;secondary: postgres-jeiaru-postgresql-0 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-verticalscaling-6mwfp ns-dpmgs VerticalScaling postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:44 UTC+0800 check ops status done ops_status:postgres-jeiaru-verticalscaling-6mwfp ns-dpmgs VerticalScaling postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 18:44 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-verticalscaling-6mwfp --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-verticalscaling-6mwfp patched `kbcli cluster delete-ops --name postgres-jeiaru-verticalscaling-6mwfp --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-verticalscaling-6mwfp deleted `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-0 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-postgres-jeiaru" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-1 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-postgres-jeiaru.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-postgres-jeiaru created apply test-chaos-mesh-networkdelay-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-networkdelay-postgres-jeiaru.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:45 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-1;secondary: postgres-jeiaru-postgresql-0 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-postgres-jeiaru patched check failover pod name networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-postgres-jeiaru" force deleted failover pod name:postgres-jeiaru-postgresql-1 failover networkdelay Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-0 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-postgres-jeiaru" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-1 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-postgres-jeiaru.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-postgres-jeiaru created apply test-chaos-mesh-networklossover-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-networklossover-postgres-jeiaru.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:45 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-postgres-jeiaru" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-postgres-jeiaru" not found check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover networklossover Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-postgres-jeiaru" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-postgres-jeiaru.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-postgres-jeiaru created apply test-chaos-mesh-dnserror-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-dnserror-postgres-jeiaru.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-postgres-jeiaru" force deleted dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-postgres-jeiaru patched check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover dnserror Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-postgres-jeiaru" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-postgres-jeiaru.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-postgres-jeiaru created apply test-chaos-mesh-fullcpu-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-fullcpu-postgres-jeiaru.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-postgres-jeiaru" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-postgres-jeiaru" not found check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover fullcpu Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote postgres-jeiaru --auto-approve --force=true --component postgresql --namespace ns-dpmgs ` component:postgresql OpsRequest postgres-jeiaru-switchover-b5z9z created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-switchover-b5z9z -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-switchover-b5z9z ns-dpmgs Switchover postgres-jeiaru postgresql Running 0/1 Jun 19,2025 18:59 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-1;secondary: postgres-jeiaru-postgresql-0 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-switchover-b5z9z ns-dpmgs Switchover postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 18:59 UTC+0800 check ops status done ops_status:postgres-jeiaru-switchover-b5z9z ns-dpmgs Switchover postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 18:59 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-switchover-b5z9z --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-switchover-b5z9z patched `kbcli cluster delete-ops --name postgres-jeiaru-switchover-b5z9z --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-switchover-b5z9z deleted `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-0 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success switchover pod:postgres-jeiaru-postgresql-1 switchover success `kubectl get secrets -l app.kubernetes.io/instance=postgres-jeiaru` set secret: postgres-jeiaru-postgresql-account-postgres `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:451Sq78xLa;DB_PORT:5432;DB_DATABASE:postgres `create database benchtest;` Unable to use a TTY - input is not a terminal or the right kind of file NOTICE: database "benchtest" does not exist, skipping return msg:Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) DROP DATABASE CREATE DATABASE apiVersion: v1 kind: Pod metadata: name: benchtest-postgres-jeiaru namespace: ns-dpmgs spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:pgsql,host:postgres-jeiaru-postgresql-postgresql.ns-dpmgs.svc.cluster.local,user:postgres,password:451Sq78xLa,port:5432,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-postgres-jeiaru.yaml` pod/benchtest-postgres-jeiaru created apply benchtest-postgres-jeiaru.yaml Success `rm -rf benchtest-postgres-jeiaru.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 0/1 ContainerCreating 0 5s pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 0/1 ContainerCreating 0 11s pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 0/1 ContainerCreating 0 17s pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 0/1 ContainerCreating 0 23s pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 1/1 Running 0 34s check pod benchtest-postgres-jeiaru status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 0/1 Completed 0 40s `kubectl logs benchtest-postgres-jeiaru --tail 30 --namespace ns-dpmgs ` [ 7s ] thds: 4 tps: 28.02 qps: 572.41 (r/w/o: 398.28/117.08/57.04) lat (ms,99%): 314.45 err/s: 1.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 28.99 qps: 586.81 (r/w/o: 414.87/111.96/59.98) lat (ms,99%): 200.47 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 28.97 qps: 601.39 (r/w/o: 419.57/120.88/60.94) lat (ms,99%): 287.38 err/s: 1.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 31.04 qps: 608.73 (r/w/o: 427.51/117.14/64.08) lat (ms,99%): 215.44 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 3780 write: 1069 other: 545 total: 5394 transactions: 267 (26.17 per sec.) queries: 5394 (528.64 per sec.) ignored errors: 3 (0.29 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.2014s total number of events: 267 Latency (ms): min: 13.30 avg: 151.65 max: 499.24 99th percentile: 397.39 sum: 40490.66 Threads fairness: events (avg/stddev): 66.7500/1.92 execution time (avg/stddev): 10.1227/0.04 `kubectl delete pod benchtest-postgres-jeiaru --force --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-postgres-jeiaru" force deleted No resources found in ns-dpmgs namespace. `kubectl get secrets -l app.kubernetes.io/instance=postgres-jeiaru` set secret: postgres-jeiaru-postgresql-account-postgres `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:451Sq78xLa;DB_PORT:5432;DB_DATABASE:postgres `create database benchtest;` Unable to use a TTY - input is not a terminal or the right kind of file return msg:Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) DROP DATABASE CREATE DATABASE apiVersion: v1 kind: Pod metadata: name: benchtest-postgres-jeiaru namespace: ns-dpmgs spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:pgsql,host:a859869e252614ba08c63a6c4e52882a-e35b22e1327633f6.elb.us-west-2.amazonaws.com,user:postgres,password:451Sq78xLa,port:5432,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-postgres-jeiaru.yaml` pod/benchtest-postgres-jeiaru created apply benchtest-postgres-jeiaru.yaml Success `rm -rf benchtest-postgres-jeiaru.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 1/1 Running 0 11s check pod benchtest-postgres-jeiaru status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-postgres-jeiaru 0/1 Completed 0 17s `kubectl logs benchtest-postgres-jeiaru --tail 30 --namespace ns-dpmgs ` [ 7s ] thds: 4 tps: 28.94 qps: 567.81 (r/w/o: 397.17/111.77/58.88) lat (ms,99%): 196.89 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 24.05 qps: 512.12 (r/w/o: 361.79/100.22/50.11) lat (ms,99%): 303.33 err/s: 1.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 16.00 qps: 317.98 (r/w/o: 220.99/65.00/32.00) lat (ms,99%): 404.61 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 24.00 qps: 484.01 (r/w/o: 341.01/95.00/48.00) lat (ms,99%): 308.84 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 3430 write: 970 other: 494 total: 4894 transactions: 242 (23.74 per sec.) queries: 4894 (480.16 per sec.) ignored errors: 3 (0.29 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.1909s total number of events: 242 Latency (ms): min: 14.52 avg: 166.84 max: 401.27 99th percentile: 397.39 sum: 40375.26 Threads fairness: events (avg/stddev): 60.5000/1.50 execution time (avg/stddev): 10.0938/0.07 `kubectl delete pod benchtest-postgres-jeiaru --force --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-postgres-jeiaru" force deleted test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-postgres-jeiaru" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-1 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-postgres-jeiaru.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-postgres-jeiaru created apply test-chaos-mesh-networkbandwidthover-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-postgres-jeiaru.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-postgres-jeiaru" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-postgres-jeiaru" not found check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover networkbandwidthover Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-postgres-jeiaru" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-postgres-jeiaru.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-postgres-jeiaru created apply test-chaos-mesh-dnsrandom-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-dnsrandom-postgres-jeiaru.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-postgres-jeiaru" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-postgres-jeiaru" not found check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover dnsrandom Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-postgres-jeiaru" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-postgres-jeiaru --namespace ns-dpmgs ` Error from server (NotFound): pods "test-db-client-connectionstress-postgres-jeiaru" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-postgres-jeiaru" not found `kubectl get secrets -l app.kubernetes.io/instance=postgres-jeiaru` set secret: postgres-jeiaru-postgresql-account-postgres `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets postgres-jeiaru-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:451Sq78xLa;DB_PORT:5432;DB_DATABASE:postgres apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-postgres-jeiaru namespace: ns-dpmgs spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "postgres-jeiaru-postgresql-postgresql.ns-dpmgs.svc.cluster.local" - "--user" - "postgres" - "--password" - "451Sq78xLa" - "--port" - "5432" - "--database" - "postgres" - "--dbtype" - "postgresql" - "--test" - "connectionstress" - "--connections" - "56" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-postgres-jeiaru.yaml` pod/test-db-client-connectionstress-postgres-jeiaru created apply test-db-client-connectionstress-postgres-jeiaru.yaml Success `rm -rf test-db-client-connectionstress-postgres-jeiaru.yaml` check pod status check pod test-db-client-connectionstress-postgres-jeiaru status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-postgres-jeiaru 0/1 Completed 0 5s check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done --host postgres-jeiaru-postgresql-postgresql.ns-dpmgs.svc.cluster.local --user postgres --password 451Sq78xLa --port 5432 --database postgres --dbtype postgresql --test connectionstress --connections 56 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 11:08:36.503 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.postgresql.jdbc.TimestampUtils (file:/app/oneclient-1.0-all.jar) to field java.util.TimeZone.defaultTimeZone WARNING: Please consider reporting this to the maintainers of org.postgresql.jdbc.TimestampUtils WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Test Result: null Connection Information: Database Type: postgresql Host: postgres-jeiaru-postgresql-postgresql.ns-dpmgs.svc.cluster.local Port: 5432 Database: postgres Table: User: postgres Org: Access Mode: mysql Test Type: connectionstress Connection Count: 56 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-postgres-jeiaru --namespace ns-dpmgs ` pod/test-db-client-connectionstress-postgres-jeiaru patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-postgres-jeiaru" force deleted check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover connectionstress Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=postgres-jeiaru,apps.kubeblocks.io/component-name=postgresql,apps.kubeblocks.io/vct-name=data --namespace ns-dpmgs ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in postgres-jeiaru namespace. `kbcli cluster volume-expand postgres-jeiaru --auto-approve --force=true --components postgresql --volume-claim-templates data --storage 7Gi --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-volumeexpansion-fn8ht created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-volumeexpansion-fn8ht -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-volumeexpansion-fn8ht ns-dpmgs VolumeExpansion postgres-jeiaru postgresql Running 0/2 Jun 19,2025 19:09 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done No resources found in postgres-jeiaru namespace. check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-volumeexpansion-fn8ht ns-dpmgs VolumeExpansion postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 19:09 UTC+0800 check ops status done ops_status:postgres-jeiaru-volumeexpansion-fn8ht ns-dpmgs VolumeExpansion postgres-jeiaru postgresql Succeed 2/2 Jun 19,2025 19:09 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-volumeexpansion-fn8ht --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-volumeexpansion-fn8ht patched `kbcli cluster delete-ops --name postgres-jeiaru-volumeexpansion-fn8ht --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-volumeexpansion-fn8ht deleted `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cmpv upgrade service version:2 set latest cmpv service version latest service version: cmpv service version upgrade and downgrade test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-postgres-jeiaru" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-postgres-jeiaru.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-postgres-jeiaru created apply test-chaos-mesh-oom-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-oom-postgres-jeiaru.yaml` check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` connect checking... check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-postgres-jeiaru" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-postgres-jeiaru patched check failover pod name failover pod name:postgres-jeiaru-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-postgres-jeiaru" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-postgres-jeiaru" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-postgres-jeiaru" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-postgres-jeiaru.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-postgres-jeiaru created apply test-chaos-mesh-oom-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-oom-postgres-jeiaru.yaml` check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-1;secondary: postgres-jeiaru-postgresql-0 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-postgres-jeiaru" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-postgres-jeiaru patched check failover pod name:postgres-jeiaru-postgresql-1 failover oom Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-0 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-postgres-jeiaru" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-1 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-postgres-jeiaru.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-postgres-jeiaru created apply test-chaos-mesh-podfailure-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-podfailure-postgres-jeiaru.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Abnormal Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-postgres-jeiaru" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-postgres-jeiaru" not found check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover podfailure Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cluster configure component_tmp: postgresql check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure postgres-jeiaru --auto-approve --force=true --set max_connections=200 --components postgresql --config-spec postgresql-configuration --config-file postgresql.conf --namespace ns-dpmgs ` Will updated configure file meta: ConfigSpec: postgresql-configuration ConfigFile: postgresql.conf ComponentName: postgresql ClusterName: postgres-jeiaru OpsRequest postgres-jeiaru-reconfiguring-2phzb created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-reconfiguring-2phzb -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-reconfiguring-2phzb ns-dpmgs Reconfiguring postgres-jeiaru postgresql Running 1/1 Jun 19,2025 19:15 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-reconfiguring-2phzb ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:15 UTC+0800 check ops status done ops_status:postgres-jeiaru-reconfiguring-2phzb ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:15 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-reconfiguring-2phzb --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-reconfiguring-2phzb patched `kbcli cluster delete-ops --name postgres-jeiaru-reconfiguring-2phzb --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-reconfiguring-2phzb deleted check config variables Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 200 configure:[max_connections] result actual:[200] equal expected:[200] `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success 12 test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-postgres-jeiaru" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-postgres-jeiaru.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-postgres-jeiaru created apply test-chaos-mesh-networkduplicate-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-networkduplicate-postgres-jeiaru.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-postgres-jeiaru" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-postgres-jeiaru" not found check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover networkduplicate Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-postgres-jeiaru" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-postgres-jeiaru.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-postgres-jeiaru created apply test-chaos-mesh-networkcorruptover-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-postgres-jeiaru.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-1;secondary: postgres-jeiaru-postgresql-0 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-postgres-jeiaru" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-postgres-jeiaru" not found check failover pod name failover pod name:postgres-jeiaru-postgresql-1 failover networkcorruptover Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-0 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-postgres-jeiaru" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-postgres-jeiaru" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-postgres-jeiaru namespace: ns-dpmgs spec: selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-1 action: partition mode: all target: mode: all selector: namespaces: - ns-dpmgs labelSelectors: apps.kubeblocks.io/pod-name: postgres-jeiaru-postgresql-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-postgres-jeiaru.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-postgres-jeiaru created apply test-chaos-mesh-networkpartition-postgres-jeiaru.yaml Success `rm -rf test-chaos-mesh-networkpartition-postgres-jeiaru.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:55 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-1;secondary: postgres-jeiaru-postgresql-0 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-postgres-jeiaru --namespace ns-dpmgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-postgres-jeiaru" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-postgres-jeiaru patched check failover pod name failover pod name:postgres-jeiaru-postgresql-1 failover networkpartition Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-1: out of postgres-jeiaru-postgresql-1(primary), postgres-jeiaru-postgresql-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-0 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success test failover delete pod:postgres-jeiaru-postgresql-1 `kubectl delete pod postgres-jeiaru-postgresql-1 --namespace ns-dpmgs ` pod "postgres-jeiaru-postgresql-1" deleted check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs Halt Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-7-10.us-west-2.compute.internal/172.31.7.10 Jun 19,2025 19:24 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check failover pod name failover pod name:postgres-jeiaru-postgresql-0 failover Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update postgres-jeiaru --termination-policy=WipeOut --namespace ns-dpmgs ` cluster.apps.kubeblocks.io/postgres-jeiaru updated check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-7-10.us-west-2.compute.internal/172.31.7.10 Jun 19,2025 19:24 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `kubectl get backupschedule -l app.kubernetes.io/instance=postgres-jeiaru ` `kubectl get backupschedule postgres-jeiaru-postgresql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched cluster pg-basebackup backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5c88d6b598-9v8l5 --namespace kb-mqtcr -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup postgres-jeiaru --method pg-basebackup --namespace ns-dpmgs ` Backup backup-ns-dpmgs-postgres-jeiaru-20250619192631 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-dpmgs-postgres-jeiaru-20250619192631 -n ns-dpmgs check backup status `kbcli cluster list-backups postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION a621c5a6-postgres-jeiaru-postg-archive-wal ns-dpmgs postgres-jeiaru archive-wal Running(AvailablePods: 1) Jun 19,2025 19:26 UTC+0800 backup-ns-dpmgs-postgres-jeiaru-20250619192631 ns-dpmgs postgres-jeiaru pg-basebackup Running Jun 19,2025 19:26 UTC+0800 backup_status:postgres-jeiaru-pg-basebackup-Running backup_status:postgres-jeiaru-pg-basebackup-Running check backup status done backup_status:backup-ns-dpmgs-postgres-jeiaru-20250619192631 ns-dpmgs postgres-jeiaru pg-basebackup Completed 11407923 12s Jun 19,2025 19:26 UTC+0800 Jun 19,2025 19:26 UTC+0800 `create table if not exists msg(id SERIAL PRIMARY KEY, msg text, time timestamp);insert into msg (msg, time) values ('kbcli-test-data-jeiaru0', now());` Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file CREATE TABLE INSERT 0 1 `insert into msg (msg, time) values ('kbcli-test-data-jeiaru1', now());` Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file INSERT 0 1 Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file pg_switch_wal --------------- 0/F5EE600 (1 row) `insert into msg (msg, time) values ('kbcli-test-data-jeiaru2', now());` Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file INSERT 0 1 Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file pg_switch_wal --------------- 0/10000108 (1 row) checking recoverable time 1 recoverable time:Jun 19,2025 19:26:49 UTC+0800 `insert into msg (msg, time) values ('kbcli-test-data-jeiaru4', now());` Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file INSERT 0 1 Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file pg_switch_wal --------------- 0/11000140 (1 row) check recoverable time 1 done recoverable time:Jun 19,2025 19:26:59 UTC+0800 cluster restore-to-time backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "postgres-jeiaru-backup" not found `kbcli cluster restore postgres-jeiaru-backup --backup a621c5a6-postgres-jeiaru-postg-archive-wal --restore-to-time "Jun 19,2025 19:26:59 UTC+0800" --namespace ns-dpmgs ` Cluster postgres-jeiaru-backup created check cluster status `kbcli cluster list postgres-jeiaru-backup --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru-backup ns-dpmgs WipeOut Jun 19,2025 19:27 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru-backup --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-backup-postgresql-0 ns-dpmgs postgres-jeiaru-backup postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-7-10.us-west-2.compute.internal/172.31.7.10 Jun 19,2025 19:27 UTC+0800 postgres-jeiaru-backup-postgresql-1 ns-dpmgs postgres-jeiaru-backup postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:27 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-backup-postgresql-0;secondary: postgres-jeiaru-backup-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs` check cluster connect done `select * from msg;` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) id | msg | time ----+-------------------------+---------------------------- 1 | kbcli-test-data-jeiaru0 | 2025-06-19 11:26:46.283791 2 | kbcli-test-data-jeiaru1 | 2025-06-19 11:26:49.29088 (2 rows) Point-In-Time Recovery Success `echo 'DROP TABLE msg;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file DROP TABLE `kubectl get backupschedule -l app.kubernetes.io/instance=postgres-jeiaru ` `kubectl get backupschedule postgres-jeiaru-postgresql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched cluster connect `echo 'create extension vector;' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs --namespace ns-dpmgs ` Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file ERROR: extension "vector" already exists Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file List of installed extensions Name | Version | Schema | Description --------------------+---------+------------+----------------------------------------------------------- file_fdw | 1.0 | public | foreign-data wrapper for flat file access pg_auth_mon | 1.1 | public | monitor connection attempts per user pg_cron | 1.5 | pg_catalog | Job scheduler for PostgreSQL pg_stat_kcache | 2.2.3 | public | Kernel statistics gathering pg_stat_statements | 1.7 | public | track execution statistics of all SQL statements executed plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language plpython3u | 1.0 | pg_catalog | PL/Python3U untrusted procedural language set_user | 3.0 | public | similar to SET ROLE but with added logging vector | 0.6.1 | public | vector data type and ivfflat and hnsw access methods (9 rows) `echo 'show max_connections;' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) max_connections ----------------- 67 (1 row) connect cluster Success set max_connections to 67 delete cluster postgres-jeiaru-backup `kbcli cluster delete postgres-jeiaru-backup --auto-approve --namespace ns-dpmgs ` Cluster postgres-jeiaru-backup deleted pod_info:postgres-jeiaru-backup-postgresql-0 5/5 Running 0 108s postgres-jeiaru-backup-postgresql-1 5/5 Running 0 108s pod_info:postgres-jeiaru-backup-postgresql-0 4/5 Terminating 0 2m9s postgres-jeiaru-backup-postgresql-1 4/5 Terminating 0 2m9s No resources found in ns-dpmgs namespace. delete cluster pod done No resources found in ns-dpmgs namespace. check cluster resource non-exist OK: pvc No resources found in ns-dpmgs namespace. delete cluster done No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups a621c5a6-postgres-jeiaru-postg-archive-wal --namespace ns-dpmgs ` backup.dataprotection.kubeblocks.io/a621c5a6-postgres-jeiaru-postg-archive-wal patched `kbcli cluster delete-backup postgres-jeiaru --name a621c5a6-postgres-jeiaru-postg-archive-wal --force --auto-approve --namespace ns-dpmgs ` Backup a621c5a6-postgres-jeiaru-postg-archive-wal deleted `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-dpmgs-postgres-jeiaru-20250619192631 --namespace ns-dpmgs ` backup.dataprotection.kubeblocks.io/backup-ns-dpmgs-postgres-jeiaru-20250619192631 patched `kbcli cluster delete-backup postgres-jeiaru --name backup-ns-dpmgs-postgres-jeiaru-20250619192631 --force --auto-approve --namespace ns-dpmgs ` Backup backup-ns-dpmgs-postgres-jeiaru-20250619192631 deleted `kubectl get backupschedule -l app.kubernetes.io/instance=postgres-jeiaru ` `kubectl get backupschedule postgres-jeiaru-postgresql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched check backup status `kbcli cluster list-backups postgres-jeiaru --namespace ns-dpmgs ` No backups found in ns-dpmgs namespace. No backups found in ns-dpmgs namespace. backup_status:-- No backups found in ns-dpmgs namespace. backup_status:-- No backups found in ns-dpmgs namespace. backup_status:-- No backups found in ns-dpmgs namespace. backup_status:-- No backups found in ns-dpmgs namespace. backup_status:-- `kubectl get backupschedule -l app.kubernetes.io/instance=postgres-jeiaru ` `kubectl get backupschedule postgres-jeiaru-postgresql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backup_status:postgres-jeiaru-pg-basebackup-Running check backup status done backup_status:postgres-jeiaru-pg-basebackup-20250619113100 ns-dpmgs postgres-jeiaru pg-basebackup Completed 11431477 11s Jun 19,2025 19:31 UTC+0800 Jun 19,2025 19:31 UTC+0800 Jun 26,2025 19:31 UTC+0800 `kubectl get backupschedule -l app.kubernetes.io/instance=postgres-jeiaru ` `kubectl get backupschedule postgres-jeiaru-postgresql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/postgres-jeiaru-postgresql-backup-schedule patched (no change) cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "postgres-jeiaru-backup" not found `kbcli cluster describe-backup postgres-jeiaru-pg-basebackup-20250619113100 --namespace ns-dpmgs ` Name: postgres-jeiaru-pg-basebackup-20250619113100 Cluster: postgres-jeiaru Namespace: ns-dpmgs Spec: Method: pg-basebackup Policy Name: postgres-jeiaru-postgresql-backup-policy Status: Phase: Completed Total Size: 11431477 ActionSet Name: postgres-basebackup Repository: backuprepo-kbcli-test Duration: 11s Expiration Time: Jun 26,2025 19:31 UTC+0800 Start Time: Jun 19,2025 19:31 UTC+0800 Completion Time: Jun 19,2025 19:31 UTC+0800 Path: /ns-dpmgs/postgres-jeiaru-a621c5a6-5589-4a1e-8238-faa730f44a27/postgresql/postgres-jeiaru-pg-basebackup-20250619113100 Time Range Start: Jun 19,2025 19:31 UTC+0800 Time Range End: Jun 19,2025 19:31 UTC+0800 Warning Events: `kbcli cluster restore postgres-jeiaru-backup --backup postgres-jeiaru-pg-basebackup-20250619113100 --namespace ns-dpmgs ` Cluster postgres-jeiaru-backup created check cluster status `kbcli cluster list postgres-jeiaru-backup --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru-backup ns-dpmgs WipeOut Jun 19,2025 19:31 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru-backup --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-backup-postgresql-0 ns-dpmgs postgres-jeiaru-backup postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:32 UTC+0800 postgres-jeiaru-backup-postgresql-1 ns-dpmgs postgres-jeiaru-backup postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:32 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-backup-postgresql-0;secondary: postgres-jeiaru-backup-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs` check cluster connect done `kbcli cluster describe-backup postgres-jeiaru-pg-basebackup-20250619113100 --namespace ns-dpmgs ` Name: postgres-jeiaru-pg-basebackup-20250619113100 Cluster: postgres-jeiaru Namespace: ns-dpmgs Spec: Method: pg-basebackup Policy Name: postgres-jeiaru-postgresql-backup-policy Status: Phase: Completed Total Size: 11431477 ActionSet Name: postgres-basebackup Repository: backuprepo-kbcli-test Duration: 11s Expiration Time: Jun 26,2025 19:31 UTC+0800 Start Time: Jun 19,2025 19:31 UTC+0800 Completion Time: Jun 19,2025 19:31 UTC+0800 Path: /ns-dpmgs/postgres-jeiaru-a621c5a6-5589-4a1e-8238-faa730f44a27/postgresql/postgres-jeiaru-pg-basebackup-20250619113100 Time Range Start: Jun 19,2025 19:31 UTC+0800 Time Range End: Jun 19,2025 19:31 UTC+0800 Warning Events: cluster connect `echo 'create extension vector;' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs --namespace ns-dpmgs ` Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file ERROR: extension "vector" already exists Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file List of installed extensions Name | Version | Schema | Description --------------------+---------+------------+----------------------------------------------------------- file_fdw | 1.0 | public | foreign-data wrapper for flat file access pg_auth_mon | 1.1 | public | monitor connection attempts per user pg_cron | 1.5 | pg_catalog | Job scheduler for PostgreSQL pg_stat_kcache | 2.2.3 | public | Kernel statistics gathering pg_stat_statements | 1.7 | public | track execution statistics of all SQL statements executed plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language plpython3u | 1.0 | pg_catalog | PL/Python3U untrusted procedural language set_user | 3.0 | public | similar to SET ROLE but with added logging vector | 0.6.1 | public | vector data type and ivfflat and hnsw access methods (9 rows) `echo 'show max_connections;' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) max_connections ----------------- 200 (1 row) connect cluster Success set max_connections to 200 delete cluster postgres-jeiaru-backup `kbcli cluster delete postgres-jeiaru-backup --auto-approve --namespace ns-dpmgs ` Cluster postgres-jeiaru-backup deleted pod_info:postgres-jeiaru-backup-postgresql-0 5/5 Running 0 77s postgres-jeiaru-backup-postgresql-1 5/5 Running 0 77s pod_info:postgres-jeiaru-backup-postgresql-0 4/5 Terminating 0 98s postgres-jeiaru-backup-postgresql-1 4/5 Terminating 0 98s No resources found in ns-dpmgs namespace. delete cluster pod done No resources found in ns-dpmgs namespace. check cluster resource non-exist OK: pvc No resources found in ns-dpmgs namespace. delete cluster done No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups postgres-jeiaru-pg-basebackup-20250619113100 --namespace ns-dpmgs ` backup.dataprotection.kubeblocks.io/postgres-jeiaru-pg-basebackup-20250619113100 patched `kbcli cluster delete-backup postgres-jeiaru --name postgres-jeiaru-pg-basebackup-20250619113100 --force --auto-approve --namespace ns-dpmgs ` Backup postgres-jeiaru-pg-basebackup-20250619113100 deleted cluster pg-basebackup backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5c88d6b598-9v8l5 --namespace kb-mqtcr -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup postgres-jeiaru --method pg-basebackup --namespace ns-dpmgs ` Backup backup-ns-dpmgs-postgres-jeiaru-20250619193428 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-dpmgs-postgres-jeiaru-20250619193428 -n ns-dpmgs check backup status `kbcli cluster list-backups postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-dpmgs-postgres-jeiaru-20250619193428 ns-dpmgs postgres-jeiaru pg-basebackup Running Jun 19,2025 19:34 UTC+0800 backup_status:postgres-jeiaru-pg-basebackup-Running backup_status:postgres-jeiaru-pg-basebackup-Running check backup status done backup_status:backup-ns-dpmgs-postgres-jeiaru-20250619193428 ns-dpmgs postgres-jeiaru pg-basebackup Completed 11431478 11s Jun 19,2025 19:34 UTC+0800 Jun 19,2025 19:34 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "postgres-jeiaru-backup" not found `kbcli cluster describe-backup backup-ns-dpmgs-postgres-jeiaru-20250619193428 --namespace ns-dpmgs ` Name: backup-ns-dpmgs-postgres-jeiaru-20250619193428 Cluster: postgres-jeiaru Namespace: ns-dpmgs Spec: Method: pg-basebackup Policy Name: postgres-jeiaru-postgresql-backup-policy Status: Phase: Completed Total Size: 11431478 ActionSet Name: postgres-basebackup Repository: backuprepo-kbcli-test Duration: 11s Start Time: Jun 19,2025 19:34 UTC+0800 Completion Time: Jun 19,2025 19:34 UTC+0800 Path: /ns-dpmgs/postgres-jeiaru-a621c5a6-5589-4a1e-8238-faa730f44a27/postgresql/backup-ns-dpmgs-postgres-jeiaru-20250619193428 Time Range Start: Jun 19,2025 19:34 UTC+0800 Time Range End: Jun 19,2025 19:34 UTC+0800 Warning Events: `kbcli cluster restore postgres-jeiaru-backup --backup backup-ns-dpmgs-postgres-jeiaru-20250619193428 --namespace ns-dpmgs ` Cluster postgres-jeiaru-backup created check cluster status `kbcli cluster list postgres-jeiaru-backup --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru-backup ns-dpmgs WipeOut Jun 19,2025 19:34 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru-backup --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-backup-postgresql-0 ns-dpmgs postgres-jeiaru-backup postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:34 UTC+0800 postgres-jeiaru-backup-postgresql-1 ns-dpmgs postgres-jeiaru-backup postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-3-13.us-west-2.compute.internal/172.31.3.13 Jun 19,2025 19:34 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-backup-postgresql-0;secondary: postgres-jeiaru-backup-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs` check cluster connect done `kbcli cluster describe-backup backup-ns-dpmgs-postgres-jeiaru-20250619193428 --namespace ns-dpmgs ` Name: backup-ns-dpmgs-postgres-jeiaru-20250619193428 Cluster: postgres-jeiaru Namespace: ns-dpmgs Spec: Method: pg-basebackup Policy Name: postgres-jeiaru-postgresql-backup-policy Status: Phase: Completed Total Size: 11431478 ActionSet Name: postgres-basebackup Repository: backuprepo-kbcli-test Duration: 11s Start Time: Jun 19,2025 19:34 UTC+0800 Completion Time: Jun 19,2025 19:34 UTC+0800 Path: /ns-dpmgs/postgres-jeiaru-a621c5a6-5589-4a1e-8238-faa730f44a27/postgresql/backup-ns-dpmgs-postgres-jeiaru-20250619193428 Time Range Start: Jun 19,2025 19:34 UTC+0800 Time Range End: Jun 19,2025 19:34 UTC+0800 Warning Events: cluster connect `echo 'create extension vector;' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs --namespace ns-dpmgs ` Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file ERROR: extension "vector" already exists Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file List of installed extensions Name | Version | Schema | Description --------------------+---------+------------+----------------------------------------------------------- file_fdw | 1.0 | public | foreign-data wrapper for flat file access pg_auth_mon | 1.1 | public | monitor connection attempts per user pg_cron | 1.5 | pg_catalog | Job scheduler for PostgreSQL pg_stat_kcache | 2.2.3 | public | Kernel statistics gathering pg_stat_statements | 1.7 | public | track execution statistics of all SQL statements executed plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language plpython3u | 1.0 | pg_catalog | PL/Python3U untrusted procedural language set_user | 3.0 | public | similar to SET ROLE but with added logging vector | 0.6.1 | public | vector data type and ivfflat and hnsw access methods (9 rows) `echo 'show max_connections;' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) max_connections ----------------- 200 (1 row) connect cluster Success set max_connections to 200 delete cluster postgres-jeiaru-backup `kbcli cluster delete postgres-jeiaru-backup --auto-approve --namespace ns-dpmgs ` Cluster postgres-jeiaru-backup deleted pod_info:postgres-jeiaru-backup-postgresql-0 5/5 Running 0 74s postgres-jeiaru-backup-postgresql-1 5/5 Running 0 74s pod_info:postgres-jeiaru-backup-postgresql-0 4/5 Terminating 0 95s postgres-jeiaru-backup-postgresql-1 4/5 Terminating 0 95s No resources found in ns-dpmgs namespace. delete cluster pod done No resources found in ns-dpmgs namespace. check cluster resource non-exist OK: pvc No resources found in ns-dpmgs namespace. delete cluster done No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. cluster rebulid instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: postgres-jeiaru-rebuildinstance- namespace: ns-dpmgs spec: type: RebuildInstance clusterName: postgres-jeiaru force: true rebuildFrom: - componentName: postgresql instances: - name: postgres-jeiaru-postgresql-1 backupName: backup-ns-dpmgs-postgres-jeiaru-20250619193428 inPlace: true check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_postgres-jeiaru.yaml` opsrequest.apps.kubeblocks.io/postgres-jeiaru-rebuildinstance-4nd5x created create test_ops_cluster_postgres-jeiaru.yaml Success `rm -rf test_ops_cluster_postgres-jeiaru.yaml` check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-postgresql-backup-schedule-7cfjq ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:26 UTC+0800 postgres-jeiaru-postgresql-backup-schedule-5hs4c ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:29 UTC+0800 postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:37 UTC+0800 check ops status done ops_status:postgres-jeiaru-rebuildinstance-4nd5x ns-dpmgs RebuildInstance postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:37 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-rebuildinstance-4nd5x --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-rebuildinstance-4nd5x patched `kbcli cluster delete-ops --name postgres-jeiaru-rebuildinstance-4nd5x --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-rebuildinstance-4nd5x deleted check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-7-10.us-west-2.compute.internal/172.31.7.10 Jun 19,2025 19:37 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-dpmgs-postgres-jeiaru-20250619193428 --namespace ns-dpmgs ` backup.dataprotection.kubeblocks.io/backup-ns-dpmgs-postgres-jeiaru-20250619193428 patched `kbcli cluster delete-backup postgres-jeiaru --name backup-ns-dpmgs-postgres-jeiaru-20250619193428 --force --auto-approve --namespace ns-dpmgs ` Backup backup-ns-dpmgs-postgres-jeiaru-20250619193428 deleted cluster volume-snapshot backup `kbcli cluster backup postgres-jeiaru --method volume-snapshot --namespace ns-dpmgs ` Backup backup-ns-dpmgs-postgres-jeiaru-20250619193849 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-dpmgs-postgres-jeiaru-20250619193849 -n ns-dpmgs check backup status `kbcli cluster list-backups postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-dpmgs-postgres-jeiaru-20250619193849 ns-dpmgs postgres-jeiaru volume-snapshot Running Jun 19,2025 19:38 UTC+0800 backup_status:postgres-jeiaru-volume-snapshot-Running backup_status:postgres-jeiaru-volume-snapshot-Running backup_status:postgres-jeiaru-volume-snapshot-Running backup_status:postgres-jeiaru-volume-snapshot-Running backup_status:postgres-jeiaru-volume-snapshot-Running backup_status:postgres-jeiaru-volume-snapshot-Running backup_status:postgres-jeiaru-volume-snapshot-Running check backup status done backup_status:backup-ns-dpmgs-postgres-jeiaru-20250619193849 ns-dpmgs postgres-jeiaru volume-snapshot Completed 7Gi 36s Jun 19,2025 19:38 UTC+0800 Jun 19,2025 19:39 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "postgres-jeiaru-backup" not found `kbcli cluster describe-backup backup-ns-dpmgs-postgres-jeiaru-20250619193849 --namespace ns-dpmgs ` Name: backup-ns-dpmgs-postgres-jeiaru-20250619193849 Cluster: postgres-jeiaru Namespace: ns-dpmgs Spec: Method: volume-snapshot Policy Name: postgres-jeiaru-postgresql-backup-policy Status: Phase: Completed Total Size: 7Gi Duration: 36s Start Time: Jun 19,2025 19:38 UTC+0800 Completion Time: Jun 19,2025 19:39 UTC+0800 Time Range Start: Jun 19,2025 19:38 UTC+0800 Time Range End: Jun 19,2025 19:38 UTC+0800 Warning Events: `kbcli cluster restore postgres-jeiaru-backup --backup backup-ns-dpmgs-postgres-jeiaru-20250619193849 --namespace ns-dpmgs ` Cluster postgres-jeiaru-backup created check cluster status `kbcli cluster list postgres-jeiaru-backup --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru-backup ns-dpmgs WipeOut Jun 19,2025 19:39 UTC+0800 cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru-backup --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-backup-postgresql-0 ns-dpmgs postgres-jeiaru-backup postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 19:39 UTC+0800 postgres-jeiaru-backup-postgresql-1 ns-dpmgs postgres-jeiaru-backup postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:39 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-backup-postgresql-0;secondary: postgres-jeiaru-backup-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs` check cluster connect done `kbcli cluster describe-backup backup-ns-dpmgs-postgres-jeiaru-20250619193849 --namespace ns-dpmgs ` Name: backup-ns-dpmgs-postgres-jeiaru-20250619193849 Cluster: postgres-jeiaru Namespace: ns-dpmgs Spec: Method: volume-snapshot Policy Name: postgres-jeiaru-postgresql-backup-policy Status: Phase: Completed Total Size: 7Gi Duration: 36s Start Time: Jun 19,2025 19:38 UTC+0800 Completion Time: Jun 19,2025 19:39 UTC+0800 Time Range Start: Jun 19,2025 19:38 UTC+0800 Time Range End: Jun 19,2025 19:38 UTC+0800 Warning Events: cluster connect `echo 'create extension vector;' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs --namespace ns-dpmgs ` Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file ERROR: extension "vector" already exists Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file List of installed extensions Name | Version | Schema | Description --------------------+---------+------------+----------------------------------------------------------- file_fdw | 1.0 | public | foreign-data wrapper for flat file access pg_auth_mon | 1.1 | public | monitor connection attempts per user pg_cron | 1.5 | pg_catalog | Job scheduler for PostgreSQL pg_stat_kcache | 2.2.3 | public | Kernel statistics gathering pg_stat_statements | 1.7 | public | track execution statistics of all SQL statements executed plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language plpython3u | 1.0 | pg_catalog | PL/Python3U untrusted procedural language set_user | 3.0 | public | similar to SET ROLE but with added logging vector | 0.6.1 | public | vector data type and ivfflat and hnsw access methods (9 rows) `echo 'show max_connections;' | kbcli cluster connect postgres-jeiaru-backup --namespace ns-dpmgs ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance postgres-jeiaru-backup-postgresql-0: out of postgres-jeiaru-backup-postgresql-0(primary), postgres-jeiaru-backup-postgresql-1(secondary) max_connections ----------------- 200 (1 row) connect cluster Success set max_connections to 200 delete cluster postgres-jeiaru-backup `kbcli cluster delete postgres-jeiaru-backup --auto-approve --namespace ns-dpmgs ` Cluster postgres-jeiaru-backup deleted pod_info:postgres-jeiaru-backup-postgresql-0 5/5 Running 0 71s postgres-jeiaru-backup-postgresql-1 5/5 Running 0 72s pod_info:postgres-jeiaru-backup-postgresql-0 4/5 Terminating 0 92s postgres-jeiaru-backup-postgresql-1 4/5 Terminating 0 93s No resources found in ns-dpmgs namespace. delete cluster pod done No resources found in ns-dpmgs namespace. check cluster resource non-exist OK: pvc No resources found in ns-dpmgs namespace. delete cluster done No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-dpmgs-postgres-jeiaru-20250619193849 --namespace ns-dpmgs ` backup.dataprotection.kubeblocks.io/backup-ns-dpmgs-postgres-jeiaru-20250619193849 patched `kbcli cluster delete-backup postgres-jeiaru --name backup-ns-dpmgs-postgres-jeiaru-20250619193849 --force --auto-approve --namespace ns-dpmgs ` Backup backup-ns-dpmgs-postgres-jeiaru-20250619193849 deleted cluster list-logs `kbcli cluster list-logs postgres-jeiaru --namespace ns-dpmgs ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update postgres-jeiaru --enable-all-logs=true --namespace ns-dpmgs Error from server (NotFound): pods "postgres-jeiaru-postgresql-0" not found cluster logs `kbcli cluster logs postgres-jeiaru --tail 30 --namespace ns-dpmgs ` Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) 2025-06-19 11:38:09.587 UTC [58] LOG ***ticks: 0, maint: 0, retry: 0*** 2025-06-19 11:38:11,205 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:38:21,208 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:38:31,202 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:38:39.614 UTC [58] LOG ***ticks: 0, maint: 0, retry: 0*** 2025-06-19 11:38:41,201 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:38:51,200 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:39:01,204 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:39:09.618 UTC [58] LOG ***ticks: 0, maint: 0, retry: 0*** 2025-06-19 11:39:11,202 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:39:21,210 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:39:31,202 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:39:39.648 UTC [58] LOG ***ticks: 0, maint: 0, retry: 0*** 2025-06-19 11:39:41,204 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:39:51,229 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:40:01,203 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:40:09.650 UTC [58] LOG ***ticks: 0, maint: 0, retry: 0*** 2025-06-19 11:40:11,273 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:40:21,283 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:40:31,203 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:40:39.678 UTC [58] LOG ***ticks: 0, maint: 0, retry: 0*** 2025-06-19 11:40:41,204 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:40:51,205 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:41:01,200 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:41:09.677 UTC [58] LOG ***ticks: 0, maint: 0, retry: 0*** 2025-06-19 11:41:11,203 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:41:21,201 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:41:31,205 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock 2025-06-19 11:41:39.707 UTC [58] LOG ***ticks: 0, maint: 0, retry: 0*** 2025-06-19 11:41:41,201 INFO: no action. I am (postgres-jeiaru-postgresql-0), the leader with the lock cluster logs running `kbcli cluster logs postgres-jeiaru --tail 30 --file-type=running --namespace ns-dpmgs ` error: get pod component definition name in cluster.yaml fail cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose postgres-jeiaru --auto-approve --force=true --type vpc --enable false --components postgresql --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-expose-8v52p created successfully, you can view the progress: kbcli cluster describe-ops postgres-jeiaru-expose-8v52p -n ns-dpmgs check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-postgresql-backup-schedule-7cfjq ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:26 UTC+0800 postgres-jeiaru-postgresql-backup-schedule-5hs4c ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:29 UTC+0800 postgres-jeiaru-expose-8v52p ns-dpmgs Expose postgres-jeiaru postgresql Running 0/1 Jun 19,2025 19:41 UTC+0800 check cluster status `kbcli cluster list postgres-jeiaru --show-labels --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS postgres-jeiaru ns-dpmgs WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=postgres-jeiaru check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances postgres-jeiaru --namespace ns-dpmgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME postgres-jeiaru-postgresql-0 ns-dpmgs postgres-jeiaru postgresql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:46 UTC+0800 postgres-jeiaru-postgresql-1 ns-dpmgs postgres-jeiaru postgresql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:7Gi ip-172-31-7-10.us-west-2.compute.internal/172.31.7.10 Jun 19,2025 19:37 UTC+0800 check pod status done check cluster role check cluster role done primary: postgres-jeiaru-postgresql-0;secondary: postgres-jeiaru-postgresql-1 check cluster connect `echo '' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs` check cluster connect done check ops status `kbcli cluster list-ops postgres-jeiaru --status all --namespace ns-dpmgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME postgres-jeiaru-postgresql-backup-schedule-7cfjq ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:26 UTC+0800 postgres-jeiaru-postgresql-backup-schedule-5hs4c ns-dpmgs Reconfiguring postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:29 UTC+0800 postgres-jeiaru-expose-8v52p ns-dpmgs Expose postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:41 UTC+0800 check ops status done ops_status:postgres-jeiaru-expose-8v52p ns-dpmgs Expose postgres-jeiaru postgresql Succeed 1/1 Jun 19,2025 19:41 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests postgres-jeiaru-expose-8v52p --namespace ns-dpmgs ` opsrequest.apps.kubeblocks.io/postgres-jeiaru-expose-8v52p patched `kbcli cluster delete-ops --name postgres-jeiaru-expose-8v52p --force --auto-approve --namespace ns-dpmgs ` OpsRequest postgres-jeiaru-expose-8v52p deleted `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect postgres-jeiaru --namespace ns-dpmgs ` check data: Connect to instance postgres-jeiaru-postgresql-0: out of postgres-jeiaru-postgresql-0(primary), postgres-jeiaru-postgresql-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster data consistent Success `echo 'SELECT value FROM tmp_table WHERE id = 1;' | kubectl exec -it postgres-jeiaru-postgresql-1 --namespace ns-dpmgs -- psql -U postgres ` check readonly data: Defaulted container "postgresql" out of: postgresql, pgbouncer, exporter, lorry, config-manager, pg-init-container (init), init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value ------- lmpkt (1 row) check cluster readonly data consistent Success check db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-0 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check db_client batch data Success check readonly db_client batch data count `echo "select count(*) from executions_loop_table;" | kubectl exec -it postgres-jeiaru-postgresql-1 -n ns-dpmgs -- psql -U postgres -d executions_loop ` check readonly db_client batch data Success delete cluster postgres-jeiaru `kbcli cluster delete postgres-jeiaru --auto-approve --namespace ns-dpmgs ` Cluster postgres-jeiaru deleted pod_info:postgres-jeiaru-postgresql-0 5/5 Running 3 (21m ago) 55m postgres-jeiaru-postgresql-1 5/5 Running 0 5m3s pod_info:postgres-jeiaru-postgresql-0 4/5 Terminating 3 (21m ago) 56m postgres-jeiaru-postgresql-1 4/5 Terminating 0 5m24s No resources found in ns-dpmgs namespace. delete cluster pod done No resources found in ns-dpmgs namespace. check cluster resource non-exist OK: pvc No resources found in ns-dpmgs namespace. delete cluster done No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. No resources found in ns-dpmgs namespace. Postgresql Test Suite All Done! --------------------------------------Postgresql (Topology = Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=postgresql;ClusterVersion=postgresql-12.14.0;]|[Description=Create a cluster with the specified cluster definition postgresql and cluster version postgresql-12.14.0] [PASSED]|[Connect]|[ComponentName=postgresql]|[Description=Connect to the cluster] [PASSED]|[AddData]|[Values=lmpkt]|[Description=Add data to the cluster] [PASSED]|[CheckAddDataReadonly]|[Values=lmpkt;Role=Readonly]|[Description=Add data to the cluster readonly] [PASSED]|[Expose]|[Enable=true;TYPE=vpc;ComponentName=postgresql]|[Description=Expose Enable the vpc service with postgresql component] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=postgresql]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Reconfiguring]|[ComponentName=postgresql;shared_buffers=512MB]|[Description=Reconfiguring the cluster specify component postgresql set shared_buffers=512MB] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=postgresql]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [PASSED]|[HorizontalScaling Out]|[ComponentName=postgresql]|[Description=HorizontalScaling Out the cluster specify component postgresql] [PASSED]|[HorizontalScaling In]|[ComponentName=postgresql]|[Description=HorizontalScaling In the cluster specify component postgresql] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=postgresql]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Update]|[Logs=Enable]|[Description=Update the cluster Logs enable] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[VerticalScaling]|[ComponentName=postgresql]|[Description=VerticalScaling the cluster specify component postgresql] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=postgresql]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=postgresql]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[SwitchOver]|[ComponentName=postgresql]|[Description=SwitchOver the cluster specify component postgresql] [PASSED]|[Bench]|[ComponentName=postgresql]|[Description=Bench the cluster service with postgresql component] [PASSED]|[Bench]|[HostType=LB;ComponentName=postgresql]|[Description=Bench the cluster LB service with postgresql component] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=postgresql]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=postgresql]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[VolumeExpansion]|[ComponentName=postgresql]|[Description=VolumeExpansion the cluster specify component postgresql] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Reconfiguring]|[ComponentName=postgresql;max_connections=200]|[Description=Reconfiguring the cluster specify component postgresql set max_connections=200] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=postgresql]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=postgresql]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=postgresql]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=postgresql]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=pg-basebackup]|[Description=The cluster pg-basebackup Backup] [PASSED]|[Restore To Time]|[BackupMethod=pg-basebackup]|[Description=The cluster pg-basebackup Restore To Time] [PASSED]|[Connect]|[ComponentName=postgresql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=pg-basebackup]|[Description=Delete the pg-basebackup restore cluster] [PASSED]|[Backup]|[Schedule=true;BackupMethod=pg-basebackup]|[Description=The cluster Schedule pg-basebackup Backup] [PASSED]|[Restore]|[Schedule=true;BackupMethod=pg-basebackup]|[Description=The cluster Schedule pg-basebackup Restore] [PASSED]|[Connect]|[ComponentName=postgresql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[Schedule=true;BackupMethod=pg-basebackup]|[Description=Delete the Schedule pg-basebackup restore cluster] [PASSED]|[Backup]|[BackupMethod=pg-basebackup]|[Description=The cluster pg-basebackup Backup] [PASSED]|[Restore]|[BackupMethod=pg-basebackup]|[Description=The cluster pg-basebackup Restore] [PASSED]|[Connect]|[ComponentName=postgresql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=pg-basebackup]|[Description=Delete the pg-basebackup restore cluster] [PASSED]|[RebuildInstance]|[ComponentName=postgresql]|[Description=Rebuild the cluster instance specify component postgresql] [PASSED]|[Backup]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Backup] [PASSED]|[Restore]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Restore] [PASSED]|[Connect]|[ComponentName=postgresql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=volume-snapshot]|[Description=Delete the volume-snapshot restore cluster] [PASSED]|[Expose]|[Disable=true;TYPE=vpc;ComponentName=postgresql]|[Description=Expose Disable the vpc service with postgresql component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]