source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-mspdq ` `kubectl create namespace ns-mspdq` namespace/ns-mspdq created create namespace ns-mspdq done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 32.1M 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 90.5M 0 --:--:-- --:--:-- --:--:-- 300M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:vmstorage set component version set component version:victoria-metrics set service versions:1.0.0 set service versions sorted:1.0.0 no cluster version found set victoria-metrics component definition set replicas first:1,1.0.0 set replicas third:1,1.0.0 set replicas fourth:1,1.0.0 set minimum cmpv service version set minimum cmpv service version replicas:1,1.0.0 REPORT_COUNT:1 CLUSTER_TOPOLOGY: set cluster topology: vm-cluster LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-mspdq namespace. termination_policy:Halt create 1 replica Halt victoria-metrics cluster check cluster definition check component definition set component definition by component version check cmpd by labels set component definition2: vm-insert by component version:victoria-metrics apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: vicmets-lkbvtk namespace: ns-mspdq spec: clusterDefinitionRef: victoria-metrics topology: vm-cluster terminationPolicy: Halt componentSpecs: - name: vmstorage serviceVersion: 1.0.0 disableExporter: true replicas: 1 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: vmselect serviceVersion: 1.0.0 disableExporter: true replicas: 1 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi - name: vminsert serviceVersion: 1.0.0 disableExporter: true replicas: 1 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi `kubectl apply -f test_create_vicmets-lkbvtk.yaml` cluster.apps.kubeblocks.io/vicmets-lkbvtk created apply test_create_vicmets-lkbvtk.yaml Success `rm -rf test_create_vicmets-lkbvtk.yaml` check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Jun 19,2025 18:13 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:13 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:13 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-10-205.us-west-2.compute.internal/172.31.10.205 Jun 19,2025 18:13 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert `kubectl get secrets -l app.kubernetes.io/instance=vicmets-lkbvtk` No resources found in ns-mspdq namespace. Not found cluster secret DB_USERNAME:;DB_PASSWORD:;DB_PORT:;DB_DATABASE: There is no password in Type: 55. describe cluster `kbcli cluster describe vicmets-lkbvtk --namespace ns-mspdq ` Name: vicmets-lkbvtk Created Time: Jun 19,2025 18:13 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-mspdq victoria-metrics Running Halt Endpoints: COMPONENT MODE INTERNAL EXTERNAL vminsert ReadWrite vicmets-lkbvtk-vminsert.ns-mspdq.svc.cluster.local:8480 vmselect ReadWrite vicmets-lkbvtk-vmselect.ns-mspdq.svc.cluster.local:8481 vmstorage ReadWrite vicmets-lkbvtk-vmstorage-0.ns-mspdq.svc.cluster.local:8482 vicmets-lkbvtk-vmstorage-0.ns-mspdq.svc.cluster.local:8401 vicmets-lkbvtk-vmstorage-0.ns-mspdq.svc.cluster.local:8400 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME vminsert vicmets-lkbvtk-vminsert-0 Running us-west-2a ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:13 UTC+0800 vmselect vicmets-lkbvtk-vmselect-0 Running us-west-2a ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:13 UTC+0800 vmstorage vicmets-lkbvtk-vmstorage-0 Running us-west-2a ip-172-31-10-205.us-west-2.compute.internal/172.31.10.205 Jun 19,2025 18:13 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS vminsert false 100m / 100m 512Mi / 512Mi vmselect false 100m / 100m 512Mi / 512Mi vmstorage false 100m / 100m 512Mi / 512Mi data:1Gi kb-default-sc Images: COMPONENT TYPE IMAGE vminsert docker.io/apecloud/vminsert:v1.101.0-cluster vmselect docker.io/apecloud/vmselect:v1.101.0-cluster vmstorage docker.io/apecloud/vmstorage:v1.101.0-cluster Show cluster events: kbcli cluster list-events -n ns-mspdq vicmets-lkbvtk `kbcli cluster label vicmets-lkbvtk app.kubernetes.io/instance- --namespace ns-mspdq ` label "app.kubernetes.io/instance" not found. `kbcli cluster label vicmets-lkbvtk app.kubernetes.io/instance=vicmets-lkbvtk --namespace ns-mspdq ` `kbcli cluster label vicmets-lkbvtk --list --namespace ns-mspdq ` NAME NAMESPACE LABELS vicmets-lkbvtk ns-mspdq app.kubernetes.io/instance=vicmets-lkbvtk clusterdefinition.kubeblocks.io/name=victoria-metrics clusterversion.kubeblocks.io/name= label cluster app.kubernetes.io/instance=vicmets-lkbvtk Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=vicmets-lkbvtk --namespace ns-mspdq ` `kbcli cluster label vicmets-lkbvtk --list --namespace ns-mspdq ` NAME NAMESPACE LABELS vicmets-lkbvtk ns-mspdq app.kubernetes.io/instance=vicmets-lkbvtk case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=victoria-metrics clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test1 Success `kbcli cluster label vicmets-lkbvtk case.name=kbcli.test2 --overwrite --namespace ns-mspdq ` `kbcli cluster label vicmets-lkbvtk --list --namespace ns-mspdq ` NAME NAMESPACE LABELS vicmets-lkbvtk ns-mspdq app.kubernetes.io/instance=vicmets-lkbvtk case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=victoria-metrics clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test2 Success `kbcli cluster label vicmets-lkbvtk case.name- --namespace ns-mspdq ` `kbcli cluster label vicmets-lkbvtk --list --namespace ns-mspdq ` NAME NAMESPACE LABELS vicmets-lkbvtk ns-mspdq app.kubernetes.io/instance=vicmets-lkbvtk clusterdefinition.kubeblocks.io/name=victoria-metrics clusterversion.kubeblocks.io/name= delete cluster label case.name Success cluster connect kbcli unsupported engine type: vmstorage cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart vicmets-lkbvtk --auto-approve --force=true --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-restart-cj44w created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-restart-cj44w -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-restart-cj44w ns-mspdq Restart vicmets-lkbvtk vminsert,vmselect,vmstorage Running 0/3 Jun 19,2025 18:14 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Updating Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-10-91.us-west-2.compute.internal/172.31.10.91 Jun 19,2025 18:14 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-10-205.us-west-2.compute.internal/172.31.10.205 Jun 19,2025 18:14 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:14 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-restart-cj44w ns-mspdq Restart vicmets-lkbvtk vminsert,vmselect,vmstorage Succeed 3/3 Jun 19,2025 18:14 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-restart-cj44w ns-mspdq Restart vicmets-lkbvtk vminsert,vmselect,vmstorage Succeed 3/3 Jun 19,2025 18:14 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-restart-cj44w --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-restart-cj44w patched `kbcli cluster delete-ops --name vicmets-lkbvtk-restart-cj44w --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-restart-cj44w deleted cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop vicmets-lkbvtk --auto-approve --force=true --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-stop-stcvt created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-stop-stcvt -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-stop-stcvt ns-mspdq Stop vicmets-lkbvtk vminsert,vmselect,vmstorage Running 0/3 Jun 19,2025 18:15 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Stopping Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-stop-stcvt ns-mspdq Stop vicmets-lkbvtk vminsert,vmselect,vmstorage Succeed 3/3 Jun 19,2025 18:15 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-stop-stcvt ns-mspdq Stop vicmets-lkbvtk vminsert,vmselect,vmstorage Succeed 3/3 Jun 19,2025 18:15 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-stop-stcvt --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-stop-stcvt patched `kbcli cluster delete-ops --name vicmets-lkbvtk-stop-stcvt --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-stop-stcvt deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start vicmets-lkbvtk --force=true --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-start-sdfls created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-start-sdfls -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-start-sdfls ns-mspdq Start vicmets-lkbvtk vminsert,vmselect,vmstorage Running 0/3 Jun 19,2025 18:16 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Updating Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-1-27.us-west-2.compute.internal/172.31.1.27 Jun 19,2025 18:16 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:16 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:16 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-start-sdfls ns-mspdq Start vicmets-lkbvtk vminsert,vmselect,vmstorage Succeed 3/3 Jun 19,2025 18:16 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-start-sdfls ns-mspdq Start vicmets-lkbvtk vminsert,vmselect,vmstorage Succeed 3/3 Jun 19,2025 18:16 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-start-sdfls --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-start-sdfls patched `kbcli cluster delete-ops --name vicmets-lkbvtk-start-sdfls --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-start-sdfls deleted `kubectl get pvc -l app.kubernetes.io/instance=vicmets-lkbvtk,apps.kubeblocks.io/component-name=vmstorage,apps.kubeblocks.io/vct-name=data --namespace ns-mspdq ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in vicmets-lkbvtk namespace. `kbcli cluster volume-expand vicmets-lkbvtk --auto-approve --force=true --components vmstorage --volume-claim-templates data --storage 6Gi --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-volumeexpansion-dlvgf created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-volumeexpansion-dlvgf -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-volumeexpansion-dlvgf ns-mspdq VolumeExpansion vicmets-lkbvtk vmstorage Creating -/- Jun 19,2025 18:16 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Updating Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-1-27.us-west-2.compute.internal/172.31.1.27 Jun 19,2025 18:16 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:16 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:6Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:16 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert No resources found in vicmets-lkbvtk namespace. check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-volumeexpansion-dlvgf ns-mspdq VolumeExpansion vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:16 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-volumeexpansion-dlvgf ns-mspdq VolumeExpansion vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:16 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-volumeexpansion-dlvgf --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-volumeexpansion-dlvgf patched `kbcli cluster delete-ops --name vicmets-lkbvtk-volumeexpansion-dlvgf --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-volumeexpansion-dlvgf deleted cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in vicmets-lkbvtk namespace. `kbcli cluster hscale vicmets-lkbvtk --auto-approve --force=true --components vmstorage --replicas 2 --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-horizontalscaling-s8646 created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-horizontalscaling-s8646 -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-horizontalscaling-s8646 ns-mspdq HorizontalScaling vicmets-lkbvtk vmstorage Running 0/1 Jun 19,2025 18:17 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Updating Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-1-27.us-west-2.compute.internal/172.31.1.27 Jun 19,2025 18:16 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:16 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:6Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:16 UTC+0800 vicmets-lkbvtk-vmstorage-1 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:6Gi ip-172-31-6-223.us-west-2.compute.internal/172.31.6.223 Jun 19,2025 18:17 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert No resources found in vicmets-lkbvtk namespace. check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-horizontalscaling-s8646 ns-mspdq HorizontalScaling vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:17 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-horizontalscaling-s8646 ns-mspdq HorizontalScaling vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:17 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-horizontalscaling-s8646 --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-horizontalscaling-s8646 patched `kbcli cluster delete-ops --name vicmets-lkbvtk-horizontalscaling-s8646 --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-horizontalscaling-s8646 deleted cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in vicmets-lkbvtk namespace. `kbcli cluster hscale vicmets-lkbvtk --auto-approve --force=true --components vmstorage --replicas 1 --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-horizontalscaling-bc5c6 created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-horizontalscaling-bc5c6 -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-horizontalscaling-bc5c6 ns-mspdq HorizontalScaling vicmets-lkbvtk vmstorage Running 0/1 Jun 19,2025 18:18 UTC+0800 ops_status:vicmets-lkbvtk-horizontalscaling-bc5c6 ns-mspdq HorizontalScaling vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:18 UTC+0800 ops HorizontalScaling Succeed or Failed Soon check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Running Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-1-27.us-west-2.compute.internal/172.31.1.27 Jun 19,2025 18:16 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 100m / 100m 512Mi / 512Mi ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:16 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:6Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:16 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert No resources found in vicmets-lkbvtk namespace. check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-horizontalscaling-bc5c6 ns-mspdq HorizontalScaling vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:18 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-horizontalscaling-bc5c6 ns-mspdq HorizontalScaling vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:18 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-horizontalscaling-bc5c6 --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-horizontalscaling-bc5c6 patched `kbcli cluster delete-ops --name vicmets-lkbvtk-horizontalscaling-bc5c6 --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-horizontalscaling-bc5c6 deleted check component vmselect exists `kubectl get components -l app.kubernetes.io/instance=vicmets-lkbvtk,apps.kubeblocks.io/component-name=vmselect --namespace ns-mspdq | (grep "vmselect" || true )` check component vminsert exists `kubectl get components -l app.kubernetes.io/instance=vicmets-lkbvtk,apps.kubeblocks.io/component-name=vminsert --namespace ns-mspdq | (grep "vminsert" || true )` cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale vicmets-lkbvtk --auto-approve --force=true --components vmselect,vminsert --cpu 200m --memory 0.6Gi --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-verticalscaling-sb26x created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-verticalscaling-sb26x -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-verticalscaling-sb26x ns-mspdq VerticalScaling vicmets-lkbvtk vmselect,vminsert Running 0/2 Jun 19,2025 18:19 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Updating Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:19 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:19 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:6Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:16 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-verticalscaling-sb26x ns-mspdq VerticalScaling vicmets-lkbvtk vmselect,vminsert Succeed 2/2 Jun 19,2025 18:19 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-verticalscaling-sb26x ns-mspdq VerticalScaling vicmets-lkbvtk vmselect,vminsert Succeed 2/2 Jun 19,2025 18:19 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-verticalscaling-sb26x --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-verticalscaling-sb26x patched `kbcli cluster delete-ops --name vicmets-lkbvtk-verticalscaling-sb26x --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-verticalscaling-sb26x deleted cluster update monitor KUBEBLOCKS NAMESPACE:kb-zkhtd get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster vicmets-lkbvtk --namespace ns-mspdq --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/vicmets-lkbvtk patched check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Running Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:19 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:19 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:6Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:16 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert create curl-jq pod vicmets-lkbvtk-kb-curl `kubectl run --namespace kb-zkhtd vicmets-lkbvtk-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/vicmets-lkbvtk-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE vicmets-lkbvtk-kb-curl 0/1 ContainerCreating 0 1s check pod vicmets-lkbvtk-kb-curl status done `kubectl wait --for=condition=Ready pod vicmets-lkbvtk-kb-curl --namespace kb-zkhtd` pod/vicmets-lkbvtk-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE vicmets-lkbvtk-kb-curl 1/1 Running 0 6s kubectl exec -it vicmets-lkbvtk-kb-curl --namespace kb-zkhtd -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it vicmets-lkbvtk-kb-curl --namespace kb-zkhtd -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 122 100 122 0 0 35610 0 --:--:-- --:--:-- --:--:-- 40666 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750328404.074,"12.699999999999998"]***]*** k8s_pod_cpu_limit: 1750328404.074 monitor cluster Success `kubectl delete pod vicmets-lkbvtk-kb-curl --namespace kb-zkhtd --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vicmets-lkbvtk-kb-curl" force deleted check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Running Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:19 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:19 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 100m / 100m 512Mi / 512Mi data:6Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:16 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale vicmets-lkbvtk --auto-approve --force=true --components vmstorage --cpu 200m --memory 0.6Gi --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-verticalscaling-w7z87 created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-verticalscaling-w7z87 -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-verticalscaling-w7z87 ns-mspdq VerticalScaling vicmets-lkbvtk vmstorage Running 0/1 Jun 19,2025 18:20 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Updating Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:19 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:19 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-10-205.us-west-2.compute.internal/172.31.10.205 Jun 19,2025 18:20 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-verticalscaling-w7z87 ns-mspdq VerticalScaling vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:20 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-verticalscaling-w7z87 ns-mspdq VerticalScaling vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:20 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-verticalscaling-w7z87 --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-verticalscaling-w7z87 patched `kbcli cluster delete-ops --name vicmets-lkbvtk-verticalscaling-w7z87 --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-verticalscaling-w7z87 deleted check component vmselect exists `kubectl get components -l app.kubernetes.io/instance=vicmets-lkbvtk,apps.kubeblocks.io/component-name=vmselect --namespace ns-mspdq | (grep "vmselect" || true )` check component vminsert exists `kubectl get components -l app.kubernetes.io/instance=vicmets-lkbvtk,apps.kubeblocks.io/component-name=vminsert --namespace ns-mspdq | (grep "vminsert" || true )` cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart vicmets-lkbvtk --auto-approve --force=true --components vmselect,vminsert --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-restart-xw76v created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-restart-xw76v -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-restart-xw76v ns-mspdq Restart vicmets-lkbvtk vmselect,vminsert Creating -/- Jun 19,2025 18:21 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Updating Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-10-205.us-west-2.compute.internal/172.31.10.205 Jun 19,2025 18:20 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-restart-xw76v ns-mspdq Restart vicmets-lkbvtk vmselect,vminsert Succeed 2/2 Jun 19,2025 18:21 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-restart-xw76v ns-mspdq Restart vicmets-lkbvtk vmselect,vminsert Succeed 2/2 Jun 19,2025 18:21 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-restart-xw76v --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-restart-xw76v patched `kbcli cluster delete-ops --name vicmets-lkbvtk-restart-xw76v --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-restart-xw76v deleted check component vmselect exists `kubectl get components -l app.kubernetes.io/instance=vicmets-lkbvtk,apps.kubeblocks.io/component-name=vmselect --namespace ns-mspdq | (grep "vmselect" || true )` check component vminsert exists `kubectl get components -l app.kubernetes.io/instance=vicmets-lkbvtk,apps.kubeblocks.io/component-name=vminsert --namespace ns-mspdq | (grep "vminsert" || true )` cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in vicmets-lkbvtk namespace. `kbcli cluster hscale vicmets-lkbvtk --auto-approve --force=true --components vmselect,vminsert --replicas 2 --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-horizontalscaling-tjlnc created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-horizontalscaling-tjlnc -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-horizontalscaling-tjlnc ns-mspdq HorizontalScaling vicmets-lkbvtk vmselect,vminsert Running 0/2 Jun 19,2025 18:21 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Updating Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vminsert-1 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmselect-1 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-7-83.us-west-2.compute.internal/172.31.7.83 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-10-205.us-west-2.compute.internal/172.31.10.205 Jun 19,2025 18:20 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert No resources found in vicmets-lkbvtk namespace. check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-horizontalscaling-tjlnc ns-mspdq HorizontalScaling vicmets-lkbvtk vmselect,vminsert Succeed 2/2 Jun 19,2025 18:21 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-horizontalscaling-tjlnc ns-mspdq HorizontalScaling vicmets-lkbvtk vmselect,vminsert Succeed 2/2 Jun 19,2025 18:21 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-horizontalscaling-tjlnc --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-horizontalscaling-tjlnc patched `kbcli cluster delete-ops --name vicmets-lkbvtk-horizontalscaling-tjlnc --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-horizontalscaling-tjlnc deleted check component vmselect exists `kubectl get components -l app.kubernetes.io/instance=vicmets-lkbvtk,apps.kubeblocks.io/component-name=vmselect --namespace ns-mspdq | (grep "vmselect" || true )` check component vminsert exists `kubectl get components -l app.kubernetes.io/instance=vicmets-lkbvtk,apps.kubeblocks.io/component-name=vminsert --namespace ns-mspdq | (grep "vminsert" || true )` cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in vicmets-lkbvtk namespace. `kbcli cluster hscale vicmets-lkbvtk --auto-approve --force=true --components vmselect,vminsert --replicas 1 --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-horizontalscaling-272f5 created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-horizontalscaling-272f5 -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-horizontalscaling-272f5 ns-mspdq HorizontalScaling vicmets-lkbvtk vmselect,vminsert Running 0/2 Jun 19,2025 18:22 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Running Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-10-205.us-west-2.compute.internal/172.31.10.205 Jun 19,2025 18:20 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert No resources found in vicmets-lkbvtk namespace. check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-horizontalscaling-272f5 ns-mspdq HorizontalScaling vicmets-lkbvtk vmselect,vminsert Succeed 2/2 Jun 19,2025 18:22 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-horizontalscaling-272f5 ns-mspdq HorizontalScaling vicmets-lkbvtk vmselect,vminsert Succeed 2/2 Jun 19,2025 18:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-horizontalscaling-272f5 --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-horizontalscaling-272f5 patched `kbcli cluster delete-ops --name vicmets-lkbvtk-horizontalscaling-272f5 --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-horizontalscaling-272f5 deleted cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart vicmets-lkbvtk --auto-approve --force=true --components vmstorage --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-restart-z58q8 created successfully, you can view the progress: kbcli cluster describe-ops vicmets-lkbvtk-restart-z58q8 -n ns-mspdq check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-restart-z58q8 ns-mspdq Restart vicmets-lkbvtk vmstorage Pending -/- Jun 19,2025 18:22 UTC+0800 ops_status:vicmets-lkbvtk-restart-z58q8 ns-mspdq Restart vicmets-lkbvtk vmstorage Pending -/- Jun 19,2025 18:22 UTC+0800 check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics Halt Updating Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-12-10.us-west-2.compute.internal/172.31.12.10 Jun 19,2025 18:22 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert check ops status `kbcli cluster list-ops vicmets-lkbvtk --status all --namespace ns-mspdq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vicmets-lkbvtk-restart-z58q8 ns-mspdq Restart vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:22 UTC+0800 check ops status done ops_status:vicmets-lkbvtk-restart-z58q8 ns-mspdq Restart vicmets-lkbvtk vmstorage Succeed 1/1 Jun 19,2025 18:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vicmets-lkbvtk-restart-z58q8 --namespace ns-mspdq ` opsrequest.apps.kubeblocks.io/vicmets-lkbvtk-restart-z58q8 patched `kbcli cluster delete-ops --name vicmets-lkbvtk-restart-z58q8 --force --auto-approve --namespace ns-mspdq ` OpsRequest vicmets-lkbvtk-restart-z58q8 deleted cluster update terminationPolicy WipeOut `kbcli cluster update vicmets-lkbvtk --termination-policy=WipeOut --namespace ns-mspdq ` cluster.apps.kubeblocks.io/vicmets-lkbvtk updated check cluster status `kbcli cluster list vicmets-lkbvtk --show-labels --namespace ns-mspdq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vicmets-lkbvtk ns-mspdq victoria-metrics WipeOut Running Jun 19,2025 18:13 UTC+0800 app.kubernetes.io/instance=vicmets-lkbvtk,clusterdefinition.kubeblocks.io/name=victoria-metrics,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vicmets-lkbvtk --namespace ns-mspdq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vicmets-lkbvtk-vminsert-0 ns-mspdq vicmets-lkbvtk vminsert Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-10-110.us-west-2.compute.internal/172.31.10.110 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmselect-0 ns-mspdq vicmets-lkbvtk vmselect Running us-west-2a 200m / 200m 644245094400m / 644245094400m ip-172-31-8-193.us-west-2.compute.internal/172.31.8.193 Jun 19,2025 18:21 UTC+0800 vicmets-lkbvtk-vmstorage-0 ns-mspdq vicmets-lkbvtk vmstorage Running us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-12-10.us-west-2.compute.internal/172.31.12.10 Jun 19,2025 18:22 UTC+0800 check pod status done connect unsupported engine type: victoria-metrics vm-insert cluster list-logs `kbcli cluster list-logs vicmets-lkbvtk --namespace ns-mspdq ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update vicmets-lkbvtk --enable-all-logs=true --namespace ns-mspdq Error from server (NotFound): pods "vicmets-lkbvtk-vmstorage-0" not found cluster logs `kbcli cluster logs vicmets-lkbvtk --tail 30 --namespace ns-mspdq ` ***"ts":"2025-06-19T10:21:09.787Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:12","msg":"build version: vminsert-20240425-150412-tags-v1.101.0-cluster-0-g310d100ed"*** ***"ts":"2025-06-19T10:21:09.787Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:13","msg":"command-line flags"*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -envflag.enable=\"true\""*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -envflag.prefix=\"VM_\""*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -httpListenAddr=\":8480\""*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -loggerFormat=\"json\""*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/lib/logger/flag.go:20","msg":" -storageNode=\"vicmets-lkbvtk-vmstorage-0:8400\""*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/app/vminsert/main.go:101","msg":"initializing netstorage for storageNodes [vicmets-lkbvtk-vmstorage-0:8400]..."*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/lib/memory/memory.go:42","msg":"limiting caches to 386546073 bytes, leaving 257697383 bytes to the OS according to -memory.allowedPercent=60"*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/app/vminsert/main.go:120","msg":"successfully initialized netstorage in 0.000 seconds"*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:119","msg":"starting server at http://127.0.0.1:8480/"*** ***"ts":"2025-06-19T10:21:09.788Z","level":"info","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:120","msg":"pprof handlers are exposed at http://127.0.0.1:8480/debug/pprof/"*** ***"ts":"2025-06-19T10:21:09.996Z","level":"info","caller":"VictoriaMetrics/app/vminsert/netstorage/netstorage.go:281","msg":"successfully dialed -storageNode=\"vicmets-lkbvtk-vmstorage-0:8400\""*** delete cluster vicmets-lkbvtk `kbcli cluster delete vicmets-lkbvtk --auto-approve --namespace ns-mspdq ` Cluster vicmets-lkbvtk deleted pod_info:vicmets-lkbvtk-vminsert-0 0/1 Completed 0 2m43s vicmets-lkbvtk-vmselect-0 0/1 Completed 0 2m44s vicmets-lkbvtk-vmstorage-0 0/1 Completed 0 63s No resources found in ns-mspdq namespace. delete cluster pod done No resources found in ns-mspdq namespace. check cluster resource non-exist OK: pvc No resources found in ns-mspdq namespace. delete cluster done No resources found in ns-mspdq namespace. No resources found in ns-mspdq namespace. No resources found in ns-mspdq namespace. Victoria Metrics Test Suite All Done! --------------------------------------Victoria Metrics (Topology = vm-cluster Replicas 1) Test Result-------------------------------------- [PASSED]|[Create]|[Topology=vm-cluster;ComponentVersion=victoria-metrics;ServiceVersion=1.0.0;]|[Description=Create a cluster with the specified topology vm-cluster and component version victoria-metrics and service version 1.0.0] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[VolumeExpansion]|[ComponentName=vmstorage]|[Description=VolumeExpansion the cluster specify component vmstorage] [PASSED]|[HorizontalScaling Out]|[ComponentName=vmstorage]|[Description=HorizontalScaling Out the cluster specify component vmstorage] [WARNING]|[Operation]|[Succeed Or Failed Soon]|[Description=-] [PASSED]|[HorizontalScaling In]|[ComponentName=vmstorage]|[Description=HorizontalScaling In the cluster specify component vmstorage] [PASSED]|[VerticalScaling]|[ComponentName=vmselect,vminsert]|[Description=VerticalScaling the cluster specify component vmselect,vminsert] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[VerticalScaling]|[ComponentName=vmstorage]|[Description=VerticalScaling the cluster specify component vmstorage] [PASSED]|[Restart]|[ComponentName=vmselect,vminsert]|[Description=Restart the cluster specify component vmselect,vminsert] [PASSED]|[HorizontalScaling Out]|[ComponentName=vmselect,vminsert]|[Description=HorizontalScaling Out the cluster specify component vmselect,vminsert] [PASSED]|[HorizontalScaling In]|[ComponentName=vmselect,vminsert]|[Description=HorizontalScaling In the cluster specify component vmselect,vminsert] [PASSED]|[Restart]|[ComponentName=vmstorage]|[Description=Restart the cluster specify component vmstorage] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]