bash test/kbcli/test_kbcli_0.9.sh --type 24 --version 0.9.5 --generate-output true --chaos-mesh true --drain-node true --random-namespace true --region eastus --cloud-provider aks CURRENT_TEST_DIR:test/kbcli source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-doklo ` `kubectl create namespace ns-doklo` namespace/ns-doklo created create namespace ns-doklo done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.5-beta.8` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 96 32.1M 96 30.9M 0 0 173M 0 --:--:-- --:--:-- --:--:-- 173M 100 32.1M 100 32.1M 0 0 177M 0 --:--:-- --:--:-- --:--:-- 602M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.5-beta.8 done Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Kubernetes Env: v1.32.6 POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default kubeblocks version is:0.9.5 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:opensearch set component version No resources found no component version found unsupported component definition REPORT_COUNT 0:0 not found component version set replicas first:2 set replicas third:2 set minimum cmpv service version set minimum cmpv service version replicas:2 REPORT_COUNT:1 CLUSTER_TOPOLOGY: Not found topology in cluster definition opensearch LIMIT_CPU:0.5 LIMIT_MEMORY:1 storage size: 20 No resources found in ns-doklo namespace. termination_policy:Halt create 2 replica Halt opensearch cluster check cluster version check cluster definition `kbcli cluster create opensrch-tmhlxg --termination-policy=Halt --cluster-definition=opensearch --enable-all-logs=false --cluster-version=opensearch-2.7.0 --set cpu=500m,memory=1Gi,replicas=2,storage=20Gi --namespace ns-doklo ` Cluster opensrch-tmhlxg created check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 Halt Sep 01,2025 11:18 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME opensrch-tmhlxg-dashboard-0 ns-doklo opensrch-tmhlxg dashboard Running 0 1 / 1 1Gi / 1Gi data:20Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:18 UTC+0800 opensrch-tmhlxg-opensearch-0 ns-doklo opensrch-tmhlxg opensearch Running 0 500m / 500m 1Gi / 1Gi data:20Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:18 UTC+0800 opensrch-tmhlxg-opensearch-1 ns-doklo opensrch-tmhlxg opensearch Running 0 500m / 500m 1Gi / 1Gi data:20Gi aks-cicdamdpool-25950949-vmss000006/10.224.0.10 Sep 01,2025 11:23 UTC+0800 check pod status done connect unsupported engine type: opensearch `kubectl get secrets -l app.kubernetes.io/instance=opensrch-tmhlxg` `kubectl get secrets opensrch-tmhlxg-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets opensrch-tmhlxg-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets opensrch-tmhlxg-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:tmmb8vjs;DB_PORT:9200;DB_DATABASE: check pod opensrch-tmhlxg-opensearch-0 container_name opensearch exist password tmmb8vjs check pod opensrch-tmhlxg-opensearch-0 container_name opensearch-master-graceful-termination-handler exist password tmmb8vjs No container logs contain secret password. describe cluster `kbcli cluster describe opensrch-tmhlxg --namespace ns-doklo ` Name: opensrch-tmhlxg Created Time: Sep 01,2025 11:18 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-doklo opensearch opensearch-2.7.0 Running Halt Endpoints: COMPONENT MODE INTERNAL EXTERNAL opensearch ReadWrite opensrch-tmhlxg-opensearch.ns-doklo.svc.cluster.local:9200 opensrch-tmhlxg-opensearch.ns-doklo.svc.cluster.local:9300 dashboard ReadWrite opensrch-tmhlxg-dashboard.ns-doklo.svc.cluster.local:5601 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME dashboard opensrch-tmhlxg-dashboard-0 Running 0 aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:18 UTC+0800 opensearch opensrch-tmhlxg-opensearch-0 Running 0 aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:18 UTC+0800 opensearch opensrch-tmhlxg-opensearch-1 Running 0 aks-cicdamdpool-25950949-vmss000006/10.224.0.10 Sep 01,2025 11:23 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS opensearch false 500m / 500m 1Gi / 1Gi data:20Gi default dashboard false 1 / 1 1Gi / 1Gi data:20Gi default Images: COMPONENT TYPE IMAGE opensearch opensearch docker.io/apecloud/opensearch:2.7.0 dashboard dashboard docker.io/apecloud/opensearch-dashboards:2.7.0 Show cluster events: kbcli cluster list-events -n ns-doklo opensrch-tmhlxg `kbcli cluster label opensrch-tmhlxg app.kubernetes.io/instance- --namespace ns-doklo ` label "app.kubernetes.io/instance" not found. `kbcli cluster label opensrch-tmhlxg app.kubernetes.io/instance=opensrch-tmhlxg --namespace ns-doklo ` `kbcli cluster label opensrch-tmhlxg --list --namespace ns-doklo ` NAME NAMESPACE LABELS opensrch-tmhlxg ns-doklo app.kubernetes.io/instance=opensrch-tmhlxg clusterdefinition.kubeblocks.io/name=opensearch clusterversion.kubeblocks.io/name=opensearch-2.7.0 label cluster app.kubernetes.io/instance=opensrch-tmhlxg Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=opensrch-tmhlxg --namespace ns-doklo ` `kbcli cluster label opensrch-tmhlxg --list --namespace ns-doklo ` NAME NAMESPACE LABELS opensrch-tmhlxg ns-doklo app.kubernetes.io/instance=opensrch-tmhlxg case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=opensearch clusterversion.kubeblocks.io/name=opensearch-2.7.0 label cluster case.name=kbcli.test1 Success `kbcli cluster label opensrch-tmhlxg case.name=kbcli.test2 --overwrite --namespace ns-doklo ` `kbcli cluster label opensrch-tmhlxg --list --namespace ns-doklo ` NAME NAMESPACE LABELS opensrch-tmhlxg ns-doklo app.kubernetes.io/instance=opensrch-tmhlxg case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=opensearch clusterversion.kubeblocks.io/name=opensearch-2.7.0 label cluster case.name=kbcli.test2 Success `kbcli cluster label opensrch-tmhlxg case.name- --namespace ns-doklo ` `kbcli cluster label opensrch-tmhlxg --list --namespace ns-doklo ` NAME NAMESPACE LABELS opensrch-tmhlxg ns-doklo app.kubernetes.io/instance=opensrch-tmhlxg clusterdefinition.kubeblocks.io/name=opensearch clusterversion.kubeblocks.io/name=opensearch-2.7.0 delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=opensrch-tmhlxg` `kubectl get secrets opensrch-tmhlxg-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets opensrch-tmhlxg-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets opensrch-tmhlxg-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:tmmb8vjs;DB_PORT:9200;DB_DATABASE: ` echo 'curl https://opensrch-tmhlxg-opensearch:9200 -ku admin:admin ' | kubectl exec -it opensrch-tmhlxg-opensearch-0 --namespace ns-doklo -- sh ` Defaulted container "opensearch" out of: opensearch, opensearch-master-graceful-termination-handler, fsgroup-volume (init), sysctl (init) Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 583 100 583 0 0 696 0 --:--:-- --:--:-- --:--:-- 695 *** "name" : "opensrch-tmhlxg-opensearch-0", "cluster_name" : "opensrch-tmhlxg", "cluster_uuid" : "It-F08zxTm2-TWZw9CzZKw", "version" : *** "distribution" : "opensearch", "number" : "2.7.0", "build_type" : "tar", "build_hash" : "b7a6e09e492b1e965d827525f7863b366ef0e304", "build_date" : "2023-04-27T21:43:09.523336706Z", "build_snapshot" : false, "lucene_version" : "9.5.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" ***, "tagline" : "The OpenSearch Project: https://opensearch.org/" *** connect cluster Success `kubectl get pvc -l app.kubernetes.io/instance=opensrch-tmhlxg,apps.kubeblocks.io/component-name=opensearch,apps.kubeblocks.io/vct-name=data --namespace ns-doklo ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in opensrch-tmhlxg namespace. `kbcli cluster volume-expand opensrch-tmhlxg --auto-approve --force=true --components opensearch --volume-claim-templates data --storage 22Gi --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-volumeexpansion-2q5jl created successfully, you can view the progress: kbcli cluster describe-ops opensrch-tmhlxg-volumeexpansion-2q5jl -n ns-doklo check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-volumeexpansion-2q5jl ns-doklo VolumeExpansion opensrch-tmhlxg opensearch Creating -/- Sep 01,2025 11:27 UTC+0800 check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 Halt Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=opensrch-tmhlxg,clusterdefinition.kubeblocks.io/name=opensearch,clusterversion.kubeblocks.io/name=opensearch-2.7.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME opensrch-tmhlxg-dashboard-0 ns-doklo opensrch-tmhlxg dashboard Running 0 1 / 1 1Gi / 1Gi data:20Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:18 UTC+0800 opensrch-tmhlxg-opensearch-0 ns-doklo opensrch-tmhlxg opensearch Running 0 500m / 500m 1Gi / 1Gi data:22Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:18 UTC+0800 opensrch-tmhlxg-opensearch-1 ns-doklo opensrch-tmhlxg opensearch Running 0 500m / 500m 1Gi / 1Gi data:22Gi aks-cicdamdpool-25950949-vmss000006/10.224.0.10 Sep 01,2025 11:23 UTC+0800 check pod status done connect unsupported engine type: opensearch No resources found in opensrch-tmhlxg namespace. check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-volumeexpansion-2q5jl ns-doklo VolumeExpansion opensrch-tmhlxg opensearch Succeed 2/2 Sep 01,2025 11:27 UTC+0800 check ops status done ops_status:opensrch-tmhlxg-volumeexpansion-2q5jl ns-doklo VolumeExpansion opensrch-tmhlxg opensearch Succeed 2/2 Sep 01,2025 11:27 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests opensrch-tmhlxg-volumeexpansion-2q5jl --namespace ns-doklo ` opsrequest.apps.kubeblocks.io/opensrch-tmhlxg-volumeexpansion-2q5jl patched `kbcli cluster delete-ops --name opensrch-tmhlxg-volumeexpansion-2q5jl --force --auto-approve --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-volumeexpansion-2q5jl deleted check component dashboard exists `kubectl get components -l app.kubernetes.io/instance=opensrch-tmhlxg,apps.kubeblocks.io/component-name=dashboard --namespace ns-doklo | (grep "dashboard" || true )` cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale opensrch-tmhlxg --auto-approve --force=true --components dashboard --cpu 600m --memory 1.1Gi --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-verticalscaling-24ss9 created successfully, you can view the progress: kbcli cluster describe-ops opensrch-tmhlxg-verticalscaling-24ss9 -n ns-doklo check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-verticalscaling-24ss9 ns-doklo VerticalScaling opensrch-tmhlxg dashboard Pending -/- Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-verticalscaling-24ss9 ns-doklo VerticalScaling opensrch-tmhlxg dashboard Creating -/- Sep 01,2025 11:30 UTC+0800 check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 Halt Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=opensrch-tmhlxg,clusterdefinition.kubeblocks.io/name=opensearch,clusterversion.kubeblocks.io/name=opensearch-2.7.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME opensrch-tmhlxg-dashboard-0 ns-doklo opensrch-tmhlxg dashboard Running 0 600m / 600m 1181116006400m / 1181116006400m data:20Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:30 UTC+0800 opensrch-tmhlxg-opensearch-0 ns-doklo opensrch-tmhlxg opensearch Running 0 500m / 500m 1Gi / 1Gi data:22Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:18 UTC+0800 opensrch-tmhlxg-opensearch-1 ns-doklo opensrch-tmhlxg opensearch Running 0 500m / 500m 1Gi / 1Gi data:22Gi aks-cicdamdpool-25950949-vmss000006/10.224.0.10 Sep 01,2025 11:23 UTC+0800 check pod status done connect unsupported engine type: opensearch check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-verticalscaling-24ss9 ns-doklo VerticalScaling opensrch-tmhlxg dashboard Succeed 1/1 Sep 01,2025 11:30 UTC+0800 check ops status done ops_status:opensrch-tmhlxg-verticalscaling-24ss9 ns-doklo VerticalScaling opensrch-tmhlxg dashboard Succeed 1/1 Sep 01,2025 11:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests opensrch-tmhlxg-verticalscaling-24ss9 --namespace ns-doklo ` opsrequest.apps.kubeblocks.io/opensrch-tmhlxg-verticalscaling-24ss9 patched `kbcli cluster delete-ops --name opensrch-tmhlxg-verticalscaling-24ss9 --force --auto-approve --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-verticalscaling-24ss9 deleted cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart opensrch-tmhlxg --auto-approve --force=true --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-restart-f26gh created successfully, you can view the progress: kbcli cluster describe-ops opensrch-tmhlxg-restart-f26gh -n ns-doklo check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Creating -/- Sep 01,2025 11:30 UTC+0800 check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 Halt Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=opensrch-tmhlxg,clusterdefinition.kubeblocks.io/name=opensearch,clusterversion.kubeblocks.io/name=opensearch-2.7.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster opensrch-tmhlxg yaml-------------------------------------- `kubectl get cluster opensrch-tmhlxg -o yaml --namespace ns-doklo ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"opensrch-tmhlxg-restart-f26gh","type":"Restart"***]' kubeblocks.io/reconcile: "2025-09-01T03:37:08.024692706Z" creationTimestamp: "2025-09-01T03:18:15Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 3 labels: app.kubernetes.io/instance: opensrch-tmhlxg clusterdefinition.kubeblocks.io/name: opensearch clusterversion.kubeblocks.io/name: opensearch-2.7.0 name: opensrch-tmhlxg namespace: ns-doklo resourceVersion: "31790" uid: 89fa88b2-d457-43ff-b22f-ee38986adc98 spec: affinity: podAntiAffinity: Preferred tenancy: SharedNode clusterDefinitionRef: opensearch clusterVersionRef: opensearch-2.7.0 componentSpecs: - componentDefRef: opensearch disableExporter: true name: opensearch replicas: 2 resources: limits: cpu: 500m memory: 1Gi requests: cpu: 500m memory: 1Gi serviceAccountName: kb-opensrch-tmhlxg volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 22Gi - componentDefRef: dashboard disableExporter: true name: dashboard replicas: 1 resources: limits: cpu: 600m memory: 1181116006400m requests: cpu: 600m memory: 1181116006400m serviceAccountName: kb-opensrch-tmhlxg volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: Halt status: clusterDefGeneration: 2 components: dashboard: phase: Running podsReady: true podsReadyTime: "2025-09-01T03:31:47Z" opensearch: phase: Updating podsReady: false podsReadyTime: "2025-09-01T03:29:54Z" conditions: - lastTransitionTime: "2025-09-01T03:18:15Z" message: 'The operator has started the provisioning of Cluster: opensrch-tmhlxg' observedGeneration: 3 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T03:18:15Z" message: Successfully applied for resources observedGeneration: 3 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T03:30:55Z" message: 'pods are not ready in Components: [opensearch], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-09-01T03:30:55Z" message: 'pods are unavailable in Components: [opensearch], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 3 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster opensrch-tmhlxg-------------------------------------- `kubectl describe cluster opensrch-tmhlxg --namespace ns-doklo ` Name: opensrch-tmhlxg Namespace: ns-doklo Labels: app.kubernetes.io/instance=opensrch-tmhlxg clusterdefinition.kubeblocks.io/name=opensearch clusterversion.kubeblocks.io/name=opensearch-2.7.0 Annotations: kubeblocks.io/ops-request: [***"name":"opensrch-tmhlxg-restart-f26gh","type":"Restart"***] kubeblocks.io/reconcile: 2025-09-01T03:37:08.024692706Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:18:15Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 3 Resource Version: 31790 UID: 89fa88b2-d457-43ff-b22f-ee38986adc98 Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: opensearch Cluster Version Ref: opensearch-2.7.0 Component Specs: Component Def Ref: opensearch Disable Exporter: true Name: opensearch Replicas: 2 Resources: Limits: Cpu: 500m Memory: 1Gi Requests: Cpu: 500m Memory: 1Gi Service Account Name: kb-opensrch-tmhlxg Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 22Gi Component Def Ref: dashboard Disable Exporter: true Name: dashboard Replicas: 1 Resources: Limits: Cpu: 600m Memory: 1181116006400m Requests: Cpu: 600m Memory: 1181116006400m Service Account Name: kb-opensrch-tmhlxg Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: Halt Status: Cluster Def Generation: 2 Components: Dashboard: Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T03:31:47Z Opensearch: Phase: Updating Pods Ready: false Pods Ready Time: 2025-09-01T03:29:54Z Conditions: Last Transition Time: 2025-09-01T03:18:15Z Message: The operator has started the provisioning of Cluster: opensrch-tmhlxg Observed Generation: 3 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T03:18:15Z Message: Successfully applied for resources Observed Generation: 3 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T03:30:55Z Message: pods are not ready in Components: [opensearch], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-09-01T03:30:55Z Message: pods are unavailable in Components: [opensearch], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 3 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 20m (x2 over 20m) cluster-controller component is Creating Warning Unhealthy 15m (x4 over 15m) event-controller Pod opensrch-tmhlxg-opensearch-0: Startup probe failed: dial tcp 10.244.3.15:9200: connect: connection refused Warning Unhealthy 11m (x3 over 12m) event-controller Pod opensrch-tmhlxg-opensearch-1: Startup probe failed: dial tcp 10.244.1.213:9200: connect: connection refused Warning ReplicasNotReady 11m (x2 over 18m) cluster-controller pods are not ready in Components: [opensearch], refer to related component message in Cluster.status.components Warning ComponentsNotReady 11m (x2 over 18m) cluster-controller pods are unavailable in Components: [opensearch], refer to related component message in Cluster.status.components Normal Running 8m24s (x2 over 11m) cluster-controller Cluster: opensrch-tmhlxg is ready, current phase is Running Normal ClusterReady 8m24s (x2 over 11m) cluster-controller Cluster: opensrch-tmhlxg is ready, current phase is Running Warning ReplicasNotReady 8m9s cluster-controller pods are not ready in Components: [dashboard], refer to related component message in Cluster.status.components Normal ApplyResourcesSucceed 8m9s (x3 over 20m) cluster-controller Successfully applied for resources Normal ComponentPhaseTransition 8m9s (x2 over 11m) cluster-controller component is Updating Normal PreCheckSucceed 8m9s (x3 over 20m) cluster-controller The operator has started the provisioning of Cluster: opensrch-tmhlxg Warning ComponentsNotReady 8m9s cluster-controller pods are unavailable in Components: [dashboard], refer to related component message in Cluster.status.components Normal AllReplicasReady 7m27s (x3 over 11m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ComponentPhaseTransition 7m27s (x4 over 18m) cluster-controller component is Running Warning FailedAttachVolume 4m30s (x2 over 5m2s) event-controller Pod opensrch-tmhlxg-dashboard-0: AttachVolume.Attach failed for volume "pvc-3578c521-9f85-4600-8343-f0cb91075e7a" : volume attachment is being deleted Warning Unhealthy 70s (x3 over 90s) event-controller Pod opensrch-tmhlxg-opensearch-1: Startup probe failed: dial tcp 10.244.3.211:9200: connect: connection refused ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME opensrch-tmhlxg-dashboard-0 ns-doklo opensrch-tmhlxg dashboard Running 0 600m / 600m 1181116006400m / 1181116006400m data:20Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:31 UTC+0800 opensrch-tmhlxg-opensearch-0 ns-doklo opensrch-tmhlxg opensearch Init:0/2 0 500m / 500m 1Gi / 1Gi data:22Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:37 UTC+0800 opensrch-tmhlxg-opensearch-1 ns-doklo opensrch-tmhlxg opensearch Running 0 500m / 500m 1Gi / 1Gi data:22Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:31 UTC+0800 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:Init:0/2 pod_status:PodInitializing pod_status:PodInitializing pod_status:PodInitializing check pod status done check cluster status again check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Running 2/3 Sep 01,2025 11:30 UTC+0800 check ops status done ops_status:opensrch-tmhlxg-restart-f26gh ns-doklo Restart opensrch-tmhlxg opensearch,dashboard Succeed 3/3 Sep 01,2025 11:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests opensrch-tmhlxg-restart-f26gh --namespace ns-doklo ` opsrequest.apps.kubeblocks.io/opensrch-tmhlxg-restart-f26gh patched `kbcli cluster delete-ops --name opensrch-tmhlxg-restart-f26gh --force --auto-approve --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-restart-f26gh deleted cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale opensrch-tmhlxg --auto-approve --force=true --components opensearch --cpu 600m --memory 1.1Gi --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-verticalscaling-zr28l created successfully, you can view the progress: kbcli cluster describe-ops opensrch-tmhlxg-verticalscaling-zr28l -n ns-doklo check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-verticalscaling-zr28l ns-doklo VerticalScaling opensrch-tmhlxg opensearch Creating -/- Sep 01,2025 11:40 UTC+0800 check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 Halt Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=opensrch-tmhlxg,clusterdefinition.kubeblocks.io/name=opensearch,clusterversion.kubeblocks.io/name=opensearch-2.7.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME opensrch-tmhlxg-dashboard-0 ns-doklo opensrch-tmhlxg dashboard Running 0 600m / 600m 1181116006400m / 1181116006400m data:20Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:31 UTC+0800 opensrch-tmhlxg-opensearch-0 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:45 UTC+0800 opensrch-tmhlxg-opensearch-1 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: opensearch check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-verticalscaling-zr28l ns-doklo VerticalScaling opensrch-tmhlxg opensearch Succeed 2/2 Sep 01,2025 11:40 UTC+0800 check ops status done ops_status:opensrch-tmhlxg-verticalscaling-zr28l ns-doklo VerticalScaling opensrch-tmhlxg opensearch Succeed 2/2 Sep 01,2025 11:40 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests opensrch-tmhlxg-verticalscaling-zr28l --namespace ns-doklo ` opsrequest.apps.kubeblocks.io/opensrch-tmhlxg-verticalscaling-zr28l patched `kbcli cluster delete-ops --name opensrch-tmhlxg-verticalscaling-zr28l --force --auto-approve --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-verticalscaling-zr28l deleted cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop opensrch-tmhlxg --auto-approve --force=true --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-stop-rt8kp created successfully, you can view the progress: kbcli cluster describe-ops opensrch-tmhlxg-stop-rt8kp -n ns-doklo check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-stop-rt8kp ns-doklo Stop opensrch-tmhlxg Creating -/- Sep 01,2025 11:46 UTC+0800 check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 Halt Stopping Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=opensrch-tmhlxg,clusterdefinition.kubeblocks.io/name=opensearch,clusterversion.kubeblocks.io/name=opensearch-2.7.0 cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-stop-rt8kp ns-doklo Stop opensrch-tmhlxg dashboard,opensearch Succeed 3/3 Sep 01,2025 11:46 UTC+0800 check ops status done ops_status:opensrch-tmhlxg-stop-rt8kp ns-doklo Stop opensrch-tmhlxg dashboard,opensearch Succeed 3/3 Sep 01,2025 11:46 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests opensrch-tmhlxg-stop-rt8kp --namespace ns-doklo ` opsrequest.apps.kubeblocks.io/opensrch-tmhlxg-stop-rt8kp patched `kbcli cluster delete-ops --name opensrch-tmhlxg-stop-rt8kp --force --auto-approve --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-stop-rt8kp deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start opensrch-tmhlxg --force=true --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-start-fcmmr created successfully, you can view the progress: kbcli cluster describe-ops opensrch-tmhlxg-start-fcmmr -n ns-doklo check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-start-fcmmr ns-doklo Start opensrch-tmhlxg Creating -/- Sep 01,2025 11:47 UTC+0800 check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 Halt Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=opensrch-tmhlxg,clusterdefinition.kubeblocks.io/name=opensearch,clusterversion.kubeblocks.io/name=opensearch-2.7.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME opensrch-tmhlxg-dashboard-0 ns-doklo opensrch-tmhlxg dashboard Running 0 600m / 600m 1181116006400m / 1181116006400m data:20Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:47 UTC+0800 opensrch-tmhlxg-opensearch-0 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:47 UTC+0800 opensrch-tmhlxg-opensearch-1 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:48 UTC+0800 check pod status done connect unsupported engine type: opensearch check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-start-fcmmr ns-doklo Start opensrch-tmhlxg dashboard,opensearch Succeed 3/3 Sep 01,2025 11:47 UTC+0800 check ops status done ops_status:opensrch-tmhlxg-start-fcmmr ns-doklo Start opensrch-tmhlxg dashboard,opensearch Succeed 3/3 Sep 01,2025 11:47 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests opensrch-tmhlxg-start-fcmmr --namespace ns-doklo ` opsrequest.apps.kubeblocks.io/opensrch-tmhlxg-start-fcmmr patched `kbcli cluster delete-ops --name opensrch-tmhlxg-start-fcmmr --force --auto-approve --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-start-fcmmr deleted 8 cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in opensrch-tmhlxg namespace. `kbcli cluster hscale opensrch-tmhlxg --auto-approve --force=true --components opensearch --replicas 3 --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-horizontalscaling-77mnv created successfully, you can view the progress: kbcli cluster describe-ops opensrch-tmhlxg-horizontalscaling-77mnv -n ns-doklo check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-horizontalscaling-77mnv ns-doklo HorizontalScaling opensrch-tmhlxg opensearch Creating -/- Sep 01,2025 11:50 UTC+0800 check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 Halt Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=opensrch-tmhlxg,clusterdefinition.kubeblocks.io/name=opensearch,clusterversion.kubeblocks.io/name=opensearch-2.7.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME opensrch-tmhlxg-dashboard-0 ns-doklo opensrch-tmhlxg dashboard Running 0 600m / 600m 1181116006400m / 1181116006400m data:20Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:47 UTC+0800 opensrch-tmhlxg-opensearch-0 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:47 UTC+0800 opensrch-tmhlxg-opensearch-1 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:48 UTC+0800 opensrch-tmhlxg-opensearch-2 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:50 UTC+0800 check pod status done connect unsupported engine type: opensearch No resources found in opensrch-tmhlxg namespace. check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-horizontalscaling-77mnv ns-doklo HorizontalScaling opensrch-tmhlxg opensearch Succeed 1/1 Sep 01,2025 11:50 UTC+0800 check ops status done ops_status:opensrch-tmhlxg-horizontalscaling-77mnv ns-doklo HorizontalScaling opensrch-tmhlxg opensearch Succeed 1/1 Sep 01,2025 11:50 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests opensrch-tmhlxg-horizontalscaling-77mnv --namespace ns-doklo ` opsrequest.apps.kubeblocks.io/opensrch-tmhlxg-horizontalscaling-77mnv patched `kbcli cluster delete-ops --name opensrch-tmhlxg-horizontalscaling-77mnv --force --auto-approve --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-horizontalscaling-77mnv deleted cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in opensrch-tmhlxg namespace. `kbcli cluster hscale opensrch-tmhlxg --auto-approve --force=true --components opensearch --replicas 2 --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-horizontalscaling-fkknn created successfully, you can view the progress: kbcli cluster describe-ops opensrch-tmhlxg-horizontalscaling-fkknn -n ns-doklo check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-horizontalscaling-fkknn ns-doklo HorizontalScaling opensrch-tmhlxg opensearch Creating -/- Sep 01,2025 11:51 UTC+0800 check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 Halt Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=opensrch-tmhlxg,clusterdefinition.kubeblocks.io/name=opensearch,clusterversion.kubeblocks.io/name=opensearch-2.7.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME opensrch-tmhlxg-dashboard-0 ns-doklo opensrch-tmhlxg dashboard Running 0 600m / 600m 1181116006400m / 1181116006400m data:20Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:47 UTC+0800 opensrch-tmhlxg-opensearch-0 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:47 UTC+0800 opensrch-tmhlxg-opensearch-1 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:48 UTC+0800 check pod status done connect unsupported engine type: opensearch No resources found in opensrch-tmhlxg namespace. check ops status `kbcli cluster list-ops opensrch-tmhlxg --status all --namespace ns-doklo ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME opensrch-tmhlxg-horizontalscaling-fkknn ns-doklo HorizontalScaling opensrch-tmhlxg opensearch Succeed 1/1 Sep 01,2025 11:51 UTC+0800 check ops status done ops_status:opensrch-tmhlxg-horizontalscaling-fkknn ns-doklo HorizontalScaling opensrch-tmhlxg opensearch Succeed 1/1 Sep 01,2025 11:51 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests opensrch-tmhlxg-horizontalscaling-fkknn --namespace ns-doklo ` opsrequest.apps.kubeblocks.io/opensrch-tmhlxg-horizontalscaling-fkknn patched `kbcli cluster delete-ops --name opensrch-tmhlxg-horizontalscaling-fkknn --force --auto-approve --namespace ns-doklo ` OpsRequest opensrch-tmhlxg-horizontalscaling-fkknn deleted cluster update terminationPolicy WipeOut `kbcli cluster update opensrch-tmhlxg --termination-policy=WipeOut --namespace ns-doklo ` cluster.apps.kubeblocks.io/opensrch-tmhlxg updated check cluster status `kbcli cluster list opensrch-tmhlxg --show-labels --namespace ns-doklo ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS opensrch-tmhlxg ns-doklo opensearch opensearch-2.7.0 WipeOut Running Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=opensrch-tmhlxg,clusterdefinition.kubeblocks.io/name=opensearch,clusterversion.kubeblocks.io/name=opensearch-2.7.0 check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances opensrch-tmhlxg --namespace ns-doklo ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME opensrch-tmhlxg-dashboard-0 ns-doklo opensrch-tmhlxg dashboard Running 0 600m / 600m 1181116006400m / 1181116006400m data:20Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:47 UTC+0800 opensrch-tmhlxg-opensearch-0 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000007/10.224.0.5 Sep 01,2025 11:47 UTC+0800 opensrch-tmhlxg-opensearch-1 ns-doklo opensrch-tmhlxg opensearch Running 0 600m / 600m 1181116006400m / 1181116006400m data:22Gi aks-cicdamdpool-25950949-vmss000004/10.224.0.6 Sep 01,2025 11:48 UTC+0800 check pod status done connect unsupported engine type: opensearch cluster list-logs `kbcli cluster list-logs opensrch-tmhlxg --namespace ns-doklo ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update opensrch-tmhlxg --enable-all-logs=true --namespace ns-doklo Error from server (NotFound): pods "opensrch-tmhlxg-opensearch-0" not found cluster logs `kbcli cluster logs opensrch-tmhlxg --tail 30 --namespace ns-doklo ` ***"type":"log","@timestamp":"2025-09-01T03:48:09Z","tags":["info","plugins-service"],"pid":453,"message":"Plugin \"dataSourceManagement\" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"*** ***"type":"log","@timestamp":"2025-09-01T03:48:09Z","tags":["info","plugins-service"],"pid":453,"message":"Plugin \"dataSource\" is disabled."*** ***"type":"log","@timestamp":"2025-09-01T03:48:09Z","tags":["info","plugins-service"],"pid":453,"message":"Plugin \"visTypeXy\" is disabled."*** ***"type":"log","@timestamp":"2025-09-01T03:48:09Z","tags":["info","plugins-service"],"pid":453,"message":"Plugin \"mlCommonsDashboards\" is disabled."*** ***"type":"log","@timestamp":"2025-09-01T03:48:09Z","tags":["warning","config","deprecation"],"pid":453,"message":"\"cpu.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuPath\""*** ***"type":"log","@timestamp":"2025-09-01T03:48:09Z","tags":["warning","config","deprecation"],"pid":453,"message":"\"cpuacct.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuAcctPath\""*** ***"type":"log","@timestamp":"2025-09-01T03:48:09Z","tags":["warning","config","deprecation"],"pid":453,"message":"\"opensearch.requestHeadersWhitelist\" is deprecated and has been replaced by \"opensearch.requestHeadersAllowlist\""*** ***"type":"log","@timestamp":"2025-09-01T03:48:09Z","tags":["info","plugins-system"],"pid":453,"message":"Setting up [49] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,embeddable,legacyExport,expressions,data,securityAnalyticsDashboards,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,searchRelevanceDashboards,reportsDashboards,indexManagementDashboards,anomalyDetectionDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,visBuilder,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,queryWorkbenchDashboards,notificationsDashboards,bfetch,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,observabilityDashboards,discover,savedObjectsManagement,securityDashboards]"*** ***"type":"log","@timestamp":"2025-09-01T03:48:10Z","tags":["info","savedobjects-service"],"pid":453,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."*** ***"type":"log","@timestamp":"2025-09-01T03:48:10Z","tags":["error","opensearch","data"],"pid":453,"message":"[ConnectionError]: connect ECONNREFUSED 10.0.173.210:9200"*** ***"type":"log","@timestamp":"2025-09-01T03:48:10Z","tags":["error","savedobjects-service"],"pid":453,"message":"Unable to retrieve version information from OpenSearch nodes."*** ***"type":"log","@timestamp":"2025-09-01T03:48:13Z","tags":["error","opensearch","data"],"pid":453,"message":"[ConnectionError]: connect ECONNREFUSED 10.0.173.210:9200"*** ***"type":"log","@timestamp":"2025-09-01T03:48:16Z","tags":["error","opensearch","data"],"pid":453,"message":"[ConnectionError]: connect ECONNREFUSED 10.0.173.210:9200"*** ***"type":"log","@timestamp":"2025-09-01T03:48:21Z","tags":["error","opensearch","data"],"pid":453,"message":"[ConnectionError]: connect ECONNREFUSED 10.0.173.210:9200"*** ***"type":"log","@timestamp":"2025-09-01T03:48:25Z","tags":["error","opensearch","data"],"pid":453,"message":"[ConnectionError]: connect ECONNREFUSED 10.0.173.210:9200"*** ***"type":"log","@timestamp":"2025-09-01T03:48:28Z","tags":["error","opensearch","data"],"pid":453,"message":"[ConnectionError]: connect ECONNREFUSED 10.0.173.210:9200"*** ***"type":"log","@timestamp":"2025-09-01T03:48:32Z","tags":["error","opensearch","data"],"pid":453,"message":"[ConnectionError]: connect ECONNREFUSED 10.0.173.210:9200"*** ***"type":"log","@timestamp":"2025-09-01T03:48:37Z","tags":["error","opensearch","data"],"pid":453,"message":"[ConnectionError]: connect ECONNREFUSED 10.0.173.210:9200"*** ***"type":"log","@timestamp":"2025-09-01T03:48:40Z","tags":["info","savedobjects-service"],"pid":453,"message":"Starting saved objects migrations"*** ***"type":"log","@timestamp":"2025-09-01T03:48:41Z","tags":["info","plugins-system"],"pid":453,"message":"Starting [49] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,embeddable,legacyExport,expressions,data,securityAnalyticsDashboards,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,searchRelevanceDashboards,reportsDashboards,indexManagementDashboards,anomalyDetectionDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,visBuilder,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,queryWorkbenchDashboards,notificationsDashboards,bfetch,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,observabilityDashboards,discover,savedObjectsManagement,securityDashboards]"*** ***"type":"log","@timestamp":"2025-09-01T03:48:42Z","tags":["listening","info"],"pid":453,"message":"Server running at http://0.0.0.0:5601"*** ***"type":"log","@timestamp":"2025-09-01T03:48:42Z","tags":["info","http","server","OpenSearchDashboards"],"pid":453,"message":"http server running at http://0.0.0.0:5601"*** delete cluster opensrch-tmhlxg `kbcli cluster delete opensrch-tmhlxg --auto-approve --namespace ns-doklo ` Cluster opensrch-tmhlxg deleted pod_info:opensrch-tmhlxg-dashboard-0 1/1 Running 0 4m45s opensrch-tmhlxg-opensearch-0 2/2 Running 0 4m45s opensrch-tmhlxg-opensearch-1 2/2 Running 0 3m44s pod_info:opensrch-tmhlxg-dashboard-0 1/1 Terminating 0 5m5s opensrch-tmhlxg-opensearch-0 2/2 Terminating 0 5m5s opensrch-tmhlxg-opensearch-1 2/2 Terminating 0 4m4s No resources found in ns-doklo namespace. delete cluster pod done No resources found in ns-doklo namespace. check cluster resource non-exist OK: pvc No resources found in ns-doklo namespace. delete cluster done No resources found in ns-doklo namespace. No resources found in ns-doklo namespace. No resources found in ns-doklo namespace. OpenSearch Test Suite All Done! --------------------------------------OpenSearch (Topology = Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=opensearch;ClusterVersion=opensearch-2.7.0;]|[Description=Create a cluster with the specified cluster definition opensearch and cluster version opensearch-2.7.0] [PASSED]|[Connect]|[ComponentName=opensearch]|[Description=Connect to the cluster] [PASSED]|[VolumeExpansion]|[ComponentName=opensearch]|[Description=VolumeExpansion the cluster specify component opensearch] [PASSED]|[VerticalScaling]|[ComponentName=dashboard]|[Description=VerticalScaling the cluster specify component dashboard] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[VerticalScaling]|[ComponentName=opensearch]|[Description=VerticalScaling the cluster specify component opensearch] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[HorizontalScaling Out]|[ComponentName=opensearch]|[Description=HorizontalScaling Out the cluster specify component opensearch] [PASSED]|[HorizontalScaling In]|[ComponentName=opensearch]|[Description=HorizontalScaling In the cluster specify component opensearch] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]