source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-rruor ` `kubectl create namespace ns-rruor` namespace/ns-rruor created create namespace ns-rruor done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 113M 0 --:--:-- --:--:-- --:--:-- 113M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:weaviate set component version No resources found no component version found unsupported component definition not found component version set replicas first:2 set replicas third:2 set minimum cmpv service version set minimum cmpv service version replicas:2 REPORT_COUNT:1 CLUSTER_TOPOLOGY: Not found topology in cluster definition weaviate LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-rruor namespace. termination_policy:DoNotTerminate create 2 replica DoNotTerminate weaviate cluster check cluster version check cluster definition `kbcli cluster create weaviate-yaghht --termination-policy=DoNotTerminate --cluster-definition=weaviate --enable-all-logs=false --cluster-version=weaviate-1.18.0 --set cpu=100m,memory=0.5Gi,replicas=2,storage=1Gi --namespace ns-rruor ` Cluster weaviate-yaghht created check cluster status `kbcli cluster list weaviate-yaghht --show-labels --namespace ns-rruor ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS weaviate-yaghht ns-rruor weaviate weaviate-1.18.0 DoNotTerminate Jun 19,2025 18:15 UTC+0800 clusterdefinition.kubeblocks.io/name=weaviate,clusterversion.kubeblocks.io/name=weaviate-1.18.0 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances weaviate-yaghht --namespace ns-rruor ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME weaviate-yaghht-weaviate-0 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:15 UTC+0800 weaviate-yaghht-weaviate-1 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-137.us-west-2.compute.internal/172.31.5.137 Jun 19,2025 18:16 UTC+0800 check pod status done connect unsupported engine type: weaviate `kubectl get secrets -l app.kubernetes.io/instance=weaviate-yaghht` `kubectl get secrets weaviate-yaghht-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets weaviate-yaghht-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets weaviate-yaghht-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:wsz6mkhc;DB_PORT:8080;DB_DATABASE: check pod weaviate-yaghht-weaviate-0 container_name weaviate exist password wsz6mkhc No container logs contain secret password. describe cluster `kbcli cluster describe weaviate-yaghht --namespace ns-rruor ` Name: weaviate-yaghht Created Time: Jun 19,2025 18:15 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-rruor weaviate weaviate-1.18.0 Running DoNotTerminate Endpoints: COMPONENT MODE INTERNAL EXTERNAL weaviate ReadWrite weaviate-yaghht-weaviate.ns-rruor.svc.cluster.local:8080 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME weaviate weaviate-yaghht-weaviate-0 Running us-west-2a ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:15 UTC+0800 weaviate weaviate-yaghht-weaviate-1 Running us-west-2a ip-172-31-5-137.us-west-2.compute.internal/172.31.5.137 Jun 19,2025 18:16 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS weaviate false 100m / 100m 512Mi / 512Mi data:1Gi kb-default-sc Images: COMPONENT TYPE IMAGE weaviate weaviate docker.io/apecloud/weaviate:1.19.6 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-rruor weaviate-yaghht `kbcli cluster label weaviate-yaghht app.kubernetes.io/instance- --namespace ns-rruor ` label "app.kubernetes.io/instance" not found. `kbcli cluster label weaviate-yaghht app.kubernetes.io/instance=weaviate-yaghht --namespace ns-rruor ` `kbcli cluster label weaviate-yaghht --list --namespace ns-rruor ` NAME NAMESPACE LABELS weaviate-yaghht ns-rruor app.kubernetes.io/instance=weaviate-yaghht clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 label cluster app.kubernetes.io/instance=weaviate-yaghht Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=weaviate-yaghht --namespace ns-rruor ` `kbcli cluster label weaviate-yaghht --list --namespace ns-rruor ` NAME NAMESPACE LABELS weaviate-yaghht ns-rruor app.kubernetes.io/instance=weaviate-yaghht case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 label cluster case.name=kbcli.test1 Success `kbcli cluster label weaviate-yaghht case.name=kbcli.test2 --overwrite --namespace ns-rruor ` `kbcli cluster label weaviate-yaghht --list --namespace ns-rruor ` NAME NAMESPACE LABELS weaviate-yaghht ns-rruor app.kubernetes.io/instance=weaviate-yaghht case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 label cluster case.name=kbcli.test2 Success `kbcli cluster label weaviate-yaghht case.name- --namespace ns-rruor ` `kbcli cluster label weaviate-yaghht --list --namespace ns-rruor ` NAME NAMESPACE LABELS weaviate-yaghht ns-rruor app.kubernetes.io/instance=weaviate-yaghht clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 delete cluster label case.name Success Reconfiguring may cause the cluster to become unavailable after restart `kubectl get pvc -l app.kubernetes.io/instance=weaviate-yaghht,apps.kubeblocks.io/component-name=weaviate,apps.kubeblocks.io/vct-name=data --namespace ns-rruor ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in weaviate-yaghht namespace. `kbcli cluster volume-expand weaviate-yaghht --auto-approve --force=true --components weaviate --volume-claim-templates data --storage 2Gi --namespace ns-rruor ` OpsRequest weaviate-yaghht-volumeexpansion-6wcq9 created successfully, you can view the progress: kbcli cluster describe-ops weaviate-yaghht-volumeexpansion-6wcq9 -n ns-rruor check ops status `kbcli cluster list-ops weaviate-yaghht --status all --namespace ns-rruor ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME weaviate-yaghht-volumeexpansion-6wcq9 ns-rruor VolumeExpansion weaviate-yaghht weaviate Running 0/2 Jun 19,2025 18:18 UTC+0800 check cluster status `kbcli cluster list weaviate-yaghht --show-labels --namespace ns-rruor ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS weaviate-yaghht ns-rruor weaviate weaviate-1.18.0 DoNotTerminate Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=weaviate-yaghht,clusterdefinition.kubeblocks.io/name=weaviate,clusterversion.kubeblocks.io/name=weaviate-1.18.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances weaviate-yaghht --namespace ns-rruor ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME weaviate-yaghht-weaviate-0 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:15 UTC+0800 weaviate-yaghht-weaviate-1 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-5-137.us-west-2.compute.internal/172.31.5.137 Jun 19,2025 18:16 UTC+0800 check pod status done connect unsupported engine type: weaviate No resources found in weaviate-yaghht namespace. check ops status `kbcli cluster list-ops weaviate-yaghht --status all --namespace ns-rruor ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME weaviate-yaghht-volumeexpansion-6wcq9 ns-rruor VolumeExpansion weaviate-yaghht weaviate Succeed 2/2 Jun 19,2025 18:18 UTC+0800 check ops status done ops_status:weaviate-yaghht-volumeexpansion-6wcq9 ns-rruor VolumeExpansion weaviate-yaghht weaviate Succeed 2/2 Jun 19,2025 18:18 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests weaviate-yaghht-volumeexpansion-6wcq9 --namespace ns-rruor ` opsrequest.apps.kubeblocks.io/weaviate-yaghht-volumeexpansion-6wcq9 patched `kbcli cluster delete-ops --name weaviate-yaghht-volumeexpansion-6wcq9 --force --auto-approve --namespace ns-rruor ` OpsRequest weaviate-yaghht-volumeexpansion-6wcq9 deleted cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop weaviate-yaghht --auto-approve --force=true --namespace ns-rruor ` OpsRequest weaviate-yaghht-stop-6bzj2 created successfully, you can view the progress: kbcli cluster describe-ops weaviate-yaghht-stop-6bzj2 -n ns-rruor check ops status `kbcli cluster list-ops weaviate-yaghht --status all --namespace ns-rruor ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME weaviate-yaghht-stop-6bzj2 ns-rruor Stop weaviate-yaghht weaviate Running 0/2 Jun 19,2025 18:19 UTC+0800 check cluster status `kbcli cluster list weaviate-yaghht --show-labels --namespace ns-rruor ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS weaviate-yaghht ns-rruor weaviate weaviate-1.18.0 DoNotTerminate Stopping Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=weaviate-yaghht,clusterdefinition.kubeblocks.io/name=weaviate,clusterversion.kubeblocks.io/name=weaviate-1.18.0 cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances weaviate-yaghht --namespace ns-rruor ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops weaviate-yaghht --status all --namespace ns-rruor ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME weaviate-yaghht-stop-6bzj2 ns-rruor Stop weaviate-yaghht weaviate Succeed 2/2 Jun 19,2025 18:19 UTC+0800 check ops status done ops_status:weaviate-yaghht-stop-6bzj2 ns-rruor Stop weaviate-yaghht weaviate Succeed 2/2 Jun 19,2025 18:19 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests weaviate-yaghht-stop-6bzj2 --namespace ns-rruor ` opsrequest.apps.kubeblocks.io/weaviate-yaghht-stop-6bzj2 patched `kbcli cluster delete-ops --name weaviate-yaghht-stop-6bzj2 --force --auto-approve --namespace ns-rruor ` OpsRequest weaviate-yaghht-stop-6bzj2 deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start weaviate-yaghht --force=true --namespace ns-rruor ` OpsRequest weaviate-yaghht-start-jxgfr created successfully, you can view the progress: kbcli cluster describe-ops weaviate-yaghht-start-jxgfr -n ns-rruor check ops status `kbcli cluster list-ops weaviate-yaghht --status all --namespace ns-rruor ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME weaviate-yaghht-start-jxgfr ns-rruor Start weaviate-yaghht Pending -/- Jun 19,2025 18:20 UTC+0800 ops_status:weaviate-yaghht-start-jxgfr ns-rruor Start weaviate-yaghht Creating -/- Jun 19,2025 18:20 UTC+0800 check cluster status `kbcli cluster list weaviate-yaghht --show-labels --namespace ns-rruor ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS weaviate-yaghht ns-rruor weaviate weaviate-1.18.0 DoNotTerminate Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=weaviate-yaghht,clusterdefinition.kubeblocks.io/name=weaviate,clusterversion.kubeblocks.io/name=weaviate-1.18.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances weaviate-yaghht --namespace ns-rruor ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME weaviate-yaghht-weaviate-0 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-5-137.us-west-2.compute.internal/172.31.5.137 Jun 19,2025 18:20 UTC+0800 weaviate-yaghht-weaviate-1 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-12-213.us-west-2.compute.internal/172.31.12.213 Jun 19,2025 18:21 UTC+0800 check pod status done connect unsupported engine type: weaviate check ops status `kbcli cluster list-ops weaviate-yaghht --status all --namespace ns-rruor ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME weaviate-yaghht-start-jxgfr ns-rruor Start weaviate-yaghht weaviate Succeed 2/2 Jun 19,2025 18:20 UTC+0800 check ops status done ops_status:weaviate-yaghht-start-jxgfr ns-rruor Start weaviate-yaghht weaviate Succeed 2/2 Jun 19,2025 18:20 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests weaviate-yaghht-start-jxgfr --namespace ns-rruor ` opsrequest.apps.kubeblocks.io/weaviate-yaghht-start-jxgfr patched `kbcli cluster delete-ops --name weaviate-yaghht-start-jxgfr --force --auto-approve --namespace ns-rruor ` OpsRequest weaviate-yaghht-start-jxgfr deleted `kubectl get secrets -l app.kubernetes.io/instance=weaviate-yaghht` `kubectl get secrets weaviate-yaghht-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets weaviate-yaghht-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets weaviate-yaghht-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:wsz6mkhc;DB_PORT:8080;DB_DATABASE: `kubectl get secrets weaviate-yaghht-conn-credential -o jsonpath="***.data.endpoint***"` endpoints:weaviate-yaghht-weaviate:8080 create curl-jq pod weaviate-yaghht-curl-jq `kubectl run --namespace ns-rruor weaviate-yaghht-curl-jq --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/weaviate-yaghht-curl-jq created check pod status pod_status:NAME READY STATUS RESTARTS AGE weaviate-yaghht-curl-jq 0/1 ContainerCreating 0 1s check pod weaviate-yaghht-curl-jq status done `kubectl wait --for=condition=Ready pod weaviate-yaghht-curl-jq --namespace ns-rruor` pod/weaviate-yaghht-curl-jq condition met pod_status:NAME READY STATUS RESTARTS AGE weaviate-yaghht-curl-jq 1/1 Running 0 6s `curl --no-progress-meter -X POST -H "Content-Type: application/json" -d '***"class": "'Questionznzrt'"***' http://weaviate-yaghht-weaviate:8080/v1/schema` Unable to use a TTY - input is not a terminal or the right kind of file exec return message:***"class":"Questionznzrt","invertedIndexConfig":***"bm25":***"b":0.75,"k1":1.2***,"cleanupIntervalSeconds":60,"stopwords":***"additions":null,"preset":"en","removals":null***,"properties":null,"replicationConfig":***"factor":1***,"shardingConfig":***"virtualPerPhysical":128,"desiredCount":2,"actualCount":2,"desiredVirtualCount":256,"actualVirtualCount":256,"key":"_id","strategy":"hash","function":"murmur3"***,"vectorIndexConfig":***"skip":false,"cleanupIntervalSeconds":300,"maxConnections":64,"efConstruction":128,"ef":-1,"dynamicEfMin":100,"dynamicEfMax":500,"dynamicEfFactor":8,"vectorCacheMaxObjects":1000000000000,"flatSearchCutoff":40000,"distance":"cosine","pq":***"enabled":false,"bitCompression":false,"segments":0,"centroids":256,"encoder":***"type":"kmeans","distribution":"log-normal"***,"vectorIndexType":"hnsw","vectorizer":"none"*** `curl -s http://weaviate-yaghht-weaviate:8080/v1/schema | jq | (grep Questionznzrt || true )` Unable to use a TTY - input is not a terminal or the right kind of file exec return msg: "class": "Questionznzrt", schema name:[Questionznzrt] equal [Questionznzrt] `kubectl delete pod weaviate-yaghht-curl-jq --force --namespace ns-rruor ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "weaviate-yaghht-curl-jq" force deleted cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart weaviate-yaghht --auto-approve --force=true --namespace ns-rruor ` OpsRequest weaviate-yaghht-restart-w9dr5 created successfully, you can view the progress: kbcli cluster describe-ops weaviate-yaghht-restart-w9dr5 -n ns-rruor check ops status `kbcli cluster list-ops weaviate-yaghht --status all --namespace ns-rruor ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 0/2 Jun 19,2025 18:23 UTC+0800 check cluster status `kbcli cluster list weaviate-yaghht --show-labels --namespace ns-rruor ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS weaviate-yaghht ns-rruor weaviate weaviate-1.18.0 DoNotTerminate Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=weaviate-yaghht,clusterdefinition.kubeblocks.io/name=weaviate,clusterversion.kubeblocks.io/name=weaviate-1.18.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster weaviate-yaghht yaml-------------------------------------- `kubectl get cluster weaviate-yaghht -o yaml --namespace ns-rruor ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"weaviate-yaghht-restart-w9dr5","type":"Restart"***]' kubeblocks.io/reconcile: "2025-06-19T10:31:13.197000868Z" creationTimestamp: "2025-06-19T10:15:18Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 4 labels: app.kubernetes.io/instance: weaviate-yaghht clusterdefinition.kubeblocks.io/name: weaviate clusterversion.kubeblocks.io/name: weaviate-1.18.0 name: weaviate-yaghht namespace: ns-rruor resourceVersion: "35301" uid: 9bfe8f1a-796c-4834-bda8-06064842be07 spec: affinity: podAntiAffinity: Preferred tenancy: SharedNode clusterDefinitionRef: weaviate clusterVersionRef: weaviate-1.18.0 componentSpecs: - componentDefRef: weaviate disableExporter: true name: weaviate replicas: 2 resources: limits: cpu: 100m memory: 512Mi requests: cpu: 100m memory: 512Mi serviceAccountName: kb-weaviate-yaghht volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: DoNotTerminate status: clusterDefGeneration: 2 components: weaviate: phase: Updating podsReady: false podsReadyTime: "2025-06-19T10:22:01Z" conditions: - lastTransitionTime: "2025-06-19T10:15:18Z" message: 'The operator has started the provisioning of Cluster: weaviate-yaghht' observedGeneration: 4 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:15:18Z" message: Successfully applied for resources observedGeneration: 4 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 4 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster weaviate-yaghht-------------------------------------- `kubectl describe cluster weaviate-yaghht --namespace ns-rruor ` Name: weaviate-yaghht Namespace: ns-rruor Labels: app.kubernetes.io/instance=weaviate-yaghht clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 Annotations: kubeblocks.io/ops-request: [***"name":"weaviate-yaghht-restart-w9dr5","type":"Restart"***] kubeblocks.io/reconcile: 2025-06-19T10:31:13.197000868Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:18Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 4 Resource Version: 35301 UID: 9bfe8f1a-796c-4834-bda8-06064842be07 Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: weaviate Cluster Version Ref: weaviate-1.18.0 Component Specs: Component Def Ref: weaviate Disable Exporter: true Name: weaviate Replicas: 2 Resources: Limits: Cpu: 100m Memory: 512Mi Requests: Cpu: 100m Memory: 512Mi Service Account Name: kb-weaviate-yaghht Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 2Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: DoNotTerminate Status: Cluster Def Generation: 2 Components: Weaviate: Phase: Updating Pods Ready: false Pods Ready Time: 2025-06-19T10:22:01Z Conditions: Last Transition Time: 2025-06-19T10:15:18Z Message: The operator has started the provisioning of Cluster: weaviate-yaghht Observed Generation: 4 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:15:18Z Message: Successfully applied for resources Observed Generation: 4 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T10:23:37Z Message: pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T10:23:37Z Message: pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 4 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 17m cluster-controller component is Creating Normal Running 13m (x2 over 15m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 13m (x2 over 15m) cluster-controller component is Running Normal AllReplicasReady 13m (x2 over 15m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 13m (x2 over 15m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 13m cluster-controller component is Stopping Normal Stopped 12m cluster-controller Cluster: weaviate-yaghht stopped successfully. Normal ComponentPhaseTransition 12m cluster-controller component is Stopped Normal ComponentPhaseTransition 11m (x2 over 14m) cluster-controller component is Updating Normal ApplyResourcesSucceed 11m (x4 over 17m) cluster-controller Successfully applied for resources Normal PreCheckSucceed 11m (x4 over 17m) cluster-controller The operator has started the provisioning of Cluster: weaviate-yaghht Normal HorizontalScale 11m component-controller start horizontal scale component weaviate of cluster weaviate-yaghht from 0 to 2 Warning ReplicasNotReady 11m (x2 over 14m) cluster-controller pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Warning ComponentsNotReady 11m (x2 over 14m) cluster-controller pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Warning Unhealthy 7m10s (x3 over 7m30s) event-controller Pod weaviate-yaghht-weaviate-0: Readiness probe failed: HTTP probe failed with statuscode: 503 Warning Unhealthy 75s (x21 over 5m44s) event-controller Pod weaviate-yaghht-weaviate-1: Readiness probe failed: HTTP probe failed with statuscode: 503 ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances weaviate-yaghht --namespace ns-rruor ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME weaviate-yaghht-weaviate-0 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:25 UTC+0800 weaviate-yaghht-weaviate-1 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-12-213.us-west-2.compute.internal/172.31.12.213 Jun 19,2025 18:24 UTC+0800 check pod status done check ops status `kbcli cluster list-ops weaviate-yaghht --status all --namespace ns-rruor ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 ops_status:weaviate-yaghht-restart-w9dr5 ns-rruor Restart weaviate-yaghht weaviate Running 1/2 Jun 19,2025 18:23 UTC+0800 check ops status timeout --------------------------------------get cluster weaviate-yaghht yaml-------------------------------------- `kubectl get cluster weaviate-yaghht -o yaml --namespace ns-rruor ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"weaviate-yaghht-restart-w9dr5","type":"Restart"***]' kubeblocks.io/reconcile: "2025-06-19T10:36:13.205892533Z" creationTimestamp: "2025-06-19T10:15:18Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 4 labels: app.kubernetes.io/instance: weaviate-yaghht clusterdefinition.kubeblocks.io/name: weaviate clusterversion.kubeblocks.io/name: weaviate-1.18.0 name: weaviate-yaghht namespace: ns-rruor resourceVersion: "39878" uid: 9bfe8f1a-796c-4834-bda8-06064842be07 spec: affinity: podAntiAffinity: Preferred tenancy: SharedNode clusterDefinitionRef: weaviate clusterVersionRef: weaviate-1.18.0 componentSpecs: - componentDefRef: weaviate disableExporter: true name: weaviate replicas: 2 resources: limits: cpu: 100m memory: 512Mi requests: cpu: 100m memory: 512Mi serviceAccountName: kb-weaviate-yaghht volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: DoNotTerminate status: clusterDefGeneration: 2 components: weaviate: phase: Updating podsReady: false podsReadyTime: "2025-06-19T10:22:01Z" conditions: - lastTransitionTime: "2025-06-19T10:15:18Z" message: 'The operator has started the provisioning of Cluster: weaviate-yaghht' observedGeneration: 4 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:15:18Z" message: Successfully applied for resources observedGeneration: 4 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 4 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster weaviate-yaghht-------------------------------------- `kubectl describe cluster weaviate-yaghht --namespace ns-rruor ` Name: weaviate-yaghht Namespace: ns-rruor Labels: app.kubernetes.io/instance=weaviate-yaghht clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 Annotations: kubeblocks.io/ops-request: [***"name":"weaviate-yaghht-restart-w9dr5","type":"Restart"***] kubeblocks.io/reconcile: 2025-06-19T10:36:13.205892533Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:18Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 4 Resource Version: 39878 UID: 9bfe8f1a-796c-4834-bda8-06064842be07 Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: weaviate Cluster Version Ref: weaviate-1.18.0 Component Specs: Component Def Ref: weaviate Disable Exporter: true Name: weaviate Replicas: 2 Resources: Limits: Cpu: 100m Memory: 512Mi Requests: Cpu: 100m Memory: 512Mi Service Account Name: kb-weaviate-yaghht Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 2Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: DoNotTerminate Status: Cluster Def Generation: 2 Components: Weaviate: Phase: Updating Pods Ready: false Pods Ready Time: 2025-06-19T10:22:01Z Conditions: Last Transition Time: 2025-06-19T10:15:18Z Message: The operator has started the provisioning of Cluster: weaviate-yaghht Observed Generation: 4 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:15:18Z Message: Successfully applied for resources Observed Generation: 4 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T10:23:37Z Message: pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T10:23:37Z Message: pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 4 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 25m cluster-controller component is Creating Normal Running 21m (x2 over 23m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 21m (x2 over 23m) cluster-controller component is Running Normal AllReplicasReady 21m (x2 over 23m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 21m (x2 over 23m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 21m cluster-controller component is Stopping Normal Stopped 20m cluster-controller Cluster: weaviate-yaghht stopped successfully. Normal ComponentPhaseTransition 20m cluster-controller component is Stopped Normal ComponentPhaseTransition 20m (x2 over 22m) cluster-controller component is Updating Normal ApplyResourcesSucceed 20m (x4 over 25m) cluster-controller Successfully applied for resources Normal PreCheckSucceed 20m (x4 over 25m) cluster-controller The operator has started the provisioning of Cluster: weaviate-yaghht Normal HorizontalScale 20m component-controller start horizontal scale component weaviate of cluster weaviate-yaghht from 0 to 2 Warning ReplicasNotReady 20m (x2 over 22m) cluster-controller pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Warning ComponentsNotReady 20m (x2 over 22m) cluster-controller pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Warning Unhealthy 15m (x3 over 15m) event-controller Pod weaviate-yaghht-weaviate-0: Readiness probe failed: HTTP probe failed with statuscode: 503 Warning Unhealthy 4m44s (x22 over 14m) event-controller Pod weaviate-yaghht-weaviate-1: Readiness probe failed: HTTP probe failed with statuscode: 503 ------------------------------------------------------------------------------------------------------------------ --------------------------------------get opsrequest weaviate-yaghht-restart-w9dr5 yaml-------------------------------------- `kubectl get opsrequest weaviate-yaghht-restart-w9dr5 -o yaml --namespace ns-rruor ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: creationTimestamp: "2025-06-19T10:23:37Z" finalizers: - opsrequest.kubeblocks.io/finalizer generateName: weaviate-yaghht-restart- generation: 2 labels: app.kubernetes.io/instance: weaviate-yaghht app.kubernetes.io/managed-by: kubeblocks ops.kubeblocks.io/ops-type: Restart name: weaviate-yaghht-restart-w9dr5 namespace: ns-rruor ownerReferences: - apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster name: weaviate-yaghht uid: 9bfe8f1a-796c-4834-bda8-06064842be07 resourceVersion: "31244" uid: 411cbf63-10f2-42bd-96a7-540e7d81e987 spec: clusterName: weaviate-yaghht enqueueOnForce: false force: true preConditionDeadlineSeconds: 0 restart: - componentName: weaviate type: Restart status: clusterGeneration: 4 components: weaviate: phase: Updating progressDetails: - endTime: "2025-06-19T10:26:51Z" message: 'Successfully restart: Pod/weaviate-yaghht-weaviate-0 in Component: weaviate' objectKey: Pod/weaviate-yaghht-weaviate-0 startTime: "2025-06-19T10:25:28Z" status: Succeed - endTime: "2025-06-19T10:24:54Z" message: 'Start to restart: Pod/weaviate-yaghht-weaviate-1 in Component: weaviate' objectKey: Pod/weaviate-yaghht-weaviate-1 startTime: "2025-06-19T10:23:37Z" status: Processing conditions: - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'wait for the controller to process the OpsRequest: weaviate-yaghht-restart-w9dr5 in Cluster: weaviate-yaghht' reason: WaitForProgressing status: "True" type: WaitForProgressing - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'OpsRequest: weaviate-yaghht-restart-w9dr5 is validated' reason: ValidateOpsRequestPassed status: "True" type: Validated - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'Start to restart database in Cluster: weaviate-yaghht' reason: RestartStarted status: "True" type: Restarting phase: Running progress: 1/2 startTimestamp: "2025-06-19T10:23:37Z" ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe opsrequest weaviate-yaghht-restart-w9dr5-------------------------------------- `kubectl describe opsrequest weaviate-yaghht-restart-w9dr5 --namespace ns-rruor ` Name: weaviate-yaghht-restart-w9dr5 Namespace: ns-rruor Labels: app.kubernetes.io/instance=weaviate-yaghht app.kubernetes.io/managed-by=kubeblocks ops.kubeblocks.io/ops-type=Restart Annotations: API Version: apps.kubeblocks.io/v1alpha1 Kind: OpsRequest Metadata: Creation Timestamp: 2025-06-19T10:23:37Z Finalizers: opsrequest.kubeblocks.io/finalizer Generate Name: weaviate-yaghht-restart- Generation: 2 Owner References: API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Name: weaviate-yaghht UID: 9bfe8f1a-796c-4834-bda8-06064842be07 Resource Version: 31244 UID: 411cbf63-10f2-42bd-96a7-540e7d81e987 Spec: Cluster Name: weaviate-yaghht Enqueue On Force: false Force: true Pre Condition Deadline Seconds: 0 Restart: Component Name: weaviate Type: Restart Status: Cluster Generation: 4 Components: Weaviate: Phase: Updating Progress Details: End Time: 2025-06-19T10:26:51Z Message: Successfully restart: Pod/weaviate-yaghht-weaviate-0 in Component: weaviate Object Key: Pod/weaviate-yaghht-weaviate-0 Start Time: 2025-06-19T10:25:28Z Status: Succeed End Time: 2025-06-19T10:24:54Z Message: Start to restart: Pod/weaviate-yaghht-weaviate-1 in Component: weaviate Object Key: Pod/weaviate-yaghht-weaviate-1 Start Time: 2025-06-19T10:23:37Z Status: Processing Conditions: Last Transition Time: 2025-06-19T10:23:37Z Message: wait for the controller to process the OpsRequest: weaviate-yaghht-restart-w9dr5 in Cluster: weaviate-yaghht Reason: WaitForProgressing Status: True Type: WaitForProgressing Last Transition Time: 2025-06-19T10:23:37Z Message: OpsRequest: weaviate-yaghht-restart-w9dr5 is validated Reason: ValidateOpsRequestPassed Status: True Type: Validated Last Transition Time: 2025-06-19T10:23:37Z Message: Start to restart database in Cluster: weaviate-yaghht Reason: RestartStarted Status: True Type: Restarting Phase: Running Progress: 1/2 Start Timestamp: 2025-06-19T10:23:37Z Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForProgressing 17m ops-request-controller wait for the controller to process the OpsRequest: weaviate-yaghht-restart-w9dr5 in Cluster: weaviate-yaghht Normal ValidateOpsRequestPassed 17m (x2 over 17m) ops-request-controller OpsRequest: weaviate-yaghht-restart-w9dr5 is validated Normal RestartStarted 17m (x2 over 17m) ops-request-controller Start to restart database in Cluster: weaviate-yaghht Normal Succeed 16m ops-request-controller Successfully restart: Pod/weaviate-yaghht-weaviate-1 in Component: weaviate Normal Processing 15m ops-request-controller Start to restart: Pod/weaviate-yaghht-weaviate-0 in Component: weaviate Normal Processing 14m (x2 over 17m) ops-request-controller Start to restart: Pod/weaviate-yaghht-weaviate-1 in Component: weaviate Normal Succeed 14m ops-request-controller Successfully restart: Pod/weaviate-yaghht-weaviate-0 in Component: weaviate ------------------------------------------------------------------------------------------------------------------ `kubectl get jobs -l ops.kubeblocks.io/ops-name=weaviate-yaghht-restart-w9dr5 --namespace ns-rruor` No resources found in ns-rruor namespace. `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests weaviate-yaghht-restart-w9dr5 --namespace ns-rruor ` opsrequest.apps.kubeblocks.io/weaviate-yaghht-restart-w9dr5 patched `kbcli cluster delete-ops --name weaviate-yaghht-restart-w9dr5 --force --auto-approve --namespace ns-rruor ` OpsRequest weaviate-yaghht-restart-w9dr5 deleted cluster update monitor KUBEBLOCKS NAMESPACE:kb-devqu get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster weaviate-yaghht --namespace ns-rruor --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/weaviate-yaghht patched check cluster status `kbcli cluster list weaviate-yaghht --show-labels --namespace ns-rruor ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS weaviate-yaghht ns-rruor weaviate weaviate-1.18.0 DoNotTerminate Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=weaviate-yaghht,clusterdefinition.kubeblocks.io/name=weaviate,clusterversion.kubeblocks.io/name=weaviate-1.18.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster weaviate-yaghht yaml-------------------------------------- `kubectl get cluster weaviate-yaghht -o yaml --namespace ns-rruor ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T10:46:13.199999082Z" creationTimestamp: "2025-06-19T10:15:18Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 5 labels: app.kubernetes.io/instance: weaviate-yaghht clusterdefinition.kubeblocks.io/name: weaviate clusterversion.kubeblocks.io/name: weaviate-1.18.0 name: weaviate-yaghht namespace: ns-rruor resourceVersion: "47906" uid: 9bfe8f1a-796c-4834-bda8-06064842be07 spec: affinity: podAntiAffinity: Preferred tenancy: SharedNode clusterDefinitionRef: weaviate clusterVersionRef: weaviate-1.18.0 componentSpecs: - componentDefRef: weaviate disableExporter: false name: weaviate replicas: 2 resources: limits: cpu: 100m memory: 512Mi requests: cpu: 100m memory: 512Mi serviceAccountName: kb-weaviate-yaghht volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: DoNotTerminate status: clusterDefGeneration: 2 components: weaviate: phase: Updating podsReady: false podsReadyTime: "2025-06-19T10:22:01Z" conditions: - lastTransitionTime: "2025-06-19T10:15:18Z" message: 'The operator has started the provisioning of Cluster: weaviate-yaghht' observedGeneration: 5 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:15:18Z" message: Successfully applied for resources observedGeneration: 5 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 5 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster weaviate-yaghht-------------------------------------- `kubectl describe cluster weaviate-yaghht --namespace ns-rruor ` Name: weaviate-yaghht Namespace: ns-rruor Labels: app.kubernetes.io/instance=weaviate-yaghht clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 Annotations: kubeblocks.io/reconcile: 2025-06-19T10:46:13.199999082Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:18Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 5 Resource Version: 47906 UID: 9bfe8f1a-796c-4834-bda8-06064842be07 Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: weaviate Cluster Version Ref: weaviate-1.18.0 Component Specs: Component Def Ref: weaviate Disable Exporter: false Name: weaviate Replicas: 2 Resources: Limits: Cpu: 100m Memory: 512Mi Requests: Cpu: 100m Memory: 512Mi Service Account Name: kb-weaviate-yaghht Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 2Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: DoNotTerminate Status: Cluster Def Generation: 2 Components: Weaviate: Phase: Updating Pods Ready: false Pods Ready Time: 2025-06-19T10:22:01Z Conditions: Last Transition Time: 2025-06-19T10:15:18Z Message: The operator has started the provisioning of Cluster: weaviate-yaghht Observed Generation: 5 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:15:18Z Message: Successfully applied for resources Observed Generation: 5 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T10:23:37Z Message: pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T10:23:37Z Message: pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 5 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 34m cluster-controller component is Creating Normal Running 30m (x2 over 32m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 30m (x2 over 32m) cluster-controller component is Running Normal AllReplicasReady 30m (x2 over 32m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 30m (x2 over 32m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 30m cluster-controller component is Stopping Normal Stopped 29m cluster-controller Cluster: weaviate-yaghht stopped successfully. Normal ComponentPhaseTransition 29m cluster-controller component is Stopped Normal ComponentPhaseTransition 29m (x2 over 31m) cluster-controller component is Updating Normal HorizontalScale 29m component-controller start horizontal scale component weaviate of cluster weaviate-yaghht from 0 to 2 Warning ReplicasNotReady 29m (x2 over 31m) cluster-controller pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Warning ComponentsNotReady 29m (x2 over 31m) cluster-controller pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Warning Unhealthy 24m (x3 over 24m) event-controller Pod weaviate-yaghht-weaviate-0: Readiness probe failed: HTTP probe failed with statuscode: 503 Normal ApplyResourcesSucceed 8m44s (x5 over 34m) cluster-controller Successfully applied for resources Normal PreCheckSucceed 8m44s (x5 over 34m) cluster-controller The operator has started the provisioning of Cluster: weaviate-yaghht Warning Unhealthy 3m44s (x24 over 23m) event-controller Pod weaviate-yaghht-weaviate-1: Readiness probe failed: HTTP probe failed with statuscode: 503 ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances weaviate-yaghht --namespace ns-rruor ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME weaviate-yaghht-weaviate-0 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:25 UTC+0800 weaviate-yaghht-weaviate-1 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-12-213.us-west-2.compute.internal/172.31.12.213 Jun 19,2025 18:24 UTC+0800 check pod status done create curl-jq pod weaviate-yaghht-kb-curl `kubectl run --namespace kb-devqu weaviate-yaghht-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/weaviate-yaghht-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE weaviate-yaghht-kb-curl 0/1 ContainerCreating 0 1s check pod weaviate-yaghht-kb-curl status done `kubectl wait --for=condition=Ready pod weaviate-yaghht-kb-curl --namespace kb-devqu` pod/weaviate-yaghht-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE weaviate-yaghht-kb-curl 1/1 Running 0 6s kubectl exec -it weaviate-yaghht-kb-curl --namespace kb-devqu -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it weaviate-yaghht-kb-curl --namespace kb-devqu -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 122 100 122 0 0 29270 0 --:--:-- --:--:-- --:--:-- 30500 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750330212.937,"13.999999999999996"]***]*** k8s_pod_cpu_limit: 1750330212.937 monitor cluster Success `kubectl delete pod weaviate-yaghht-kb-curl --namespace kb-devqu --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "weaviate-yaghht-kb-curl" force deleted check cluster status `kbcli cluster list weaviate-yaghht --show-labels --namespace ns-rruor ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS weaviate-yaghht ns-rruor weaviate weaviate-1.18.0 DoNotTerminate Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=weaviate-yaghht,clusterdefinition.kubeblocks.io/name=weaviate,clusterversion.kubeblocks.io/name=weaviate-1.18.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster weaviate-yaghht yaml-------------------------------------- `kubectl get cluster weaviate-yaghht -o yaml --namespace ns-rruor ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T10:56:14.193394706Z" creationTimestamp: "2025-06-19T10:15:18Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 5 labels: app.kubernetes.io/instance: weaviate-yaghht clusterdefinition.kubeblocks.io/name: weaviate clusterversion.kubeblocks.io/name: weaviate-1.18.0 name: weaviate-yaghht namespace: ns-rruor resourceVersion: "55584" uid: 9bfe8f1a-796c-4834-bda8-06064842be07 spec: affinity: podAntiAffinity: Preferred tenancy: SharedNode clusterDefinitionRef: weaviate clusterVersionRef: weaviate-1.18.0 componentSpecs: - componentDefRef: weaviate disableExporter: false name: weaviate replicas: 2 resources: limits: cpu: 100m memory: 512Mi requests: cpu: 100m memory: 512Mi serviceAccountName: kb-weaviate-yaghht volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: DoNotTerminate status: clusterDefGeneration: 2 components: weaviate: phase: Updating podsReady: false podsReadyTime: "2025-06-19T10:22:01Z" conditions: - lastTransitionTime: "2025-06-19T10:15:18Z" message: 'The operator has started the provisioning of Cluster: weaviate-yaghht' observedGeneration: 5 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:15:18Z" message: Successfully applied for resources observedGeneration: 5 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 5 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster weaviate-yaghht-------------------------------------- `kubectl describe cluster weaviate-yaghht --namespace ns-rruor ` Name: weaviate-yaghht Namespace: ns-rruor Labels: app.kubernetes.io/instance=weaviate-yaghht clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 Annotations: kubeblocks.io/reconcile: 2025-06-19T10:56:14.193394706Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:18Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 5 Resource Version: 55584 UID: 9bfe8f1a-796c-4834-bda8-06064842be07 Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: weaviate Cluster Version Ref: weaviate-1.18.0 Component Specs: Component Def Ref: weaviate Disable Exporter: false Name: weaviate Replicas: 2 Resources: Limits: Cpu: 100m Memory: 512Mi Requests: Cpu: 100m Memory: 512Mi Service Account Name: kb-weaviate-yaghht Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 2Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: DoNotTerminate Status: Cluster Def Generation: 2 Components: Weaviate: Phase: Updating Pods Ready: false Pods Ready Time: 2025-06-19T10:22:01Z Conditions: Last Transition Time: 2025-06-19T10:15:18Z Message: The operator has started the provisioning of Cluster: weaviate-yaghht Observed Generation: 5 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:15:18Z Message: Successfully applied for resources Observed Generation: 5 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T10:23:37Z Message: pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T10:23:37Z Message: pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 5 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 43m cluster-controller component is Creating Normal Running 39m (x2 over 41m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 39m (x2 over 41m) cluster-controller component is Running Normal AllReplicasReady 39m (x2 over 41m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 39m (x2 over 41m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 39m cluster-controller component is Stopping Normal Stopped 38m cluster-controller Cluster: weaviate-yaghht stopped successfully. Normal ComponentPhaseTransition 38m cluster-controller component is Stopped Normal ComponentPhaseTransition 38m (x2 over 40m) cluster-controller component is Updating Normal HorizontalScale 38m component-controller start horizontal scale component weaviate of cluster weaviate-yaghht from 0 to 2 Warning ReplicasNotReady 38m (x2 over 40m) cluster-controller pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Warning ComponentsNotReady 38m (x2 over 40m) cluster-controller pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Warning Unhealthy 33m (x3 over 34m) event-controller Pod weaviate-yaghht-weaviate-0: Readiness probe failed: HTTP probe failed with statuscode: 503 Normal ApplyResourcesSucceed 17m (x5 over 43m) cluster-controller Successfully applied for resources Normal PreCheckSucceed 17m (x5 over 43m) cluster-controller The operator has started the provisioning of Cluster: weaviate-yaghht Warning Unhealthy 2m44s (x26 over 32m) event-controller Pod weaviate-yaghht-weaviate-1: Readiness probe failed: HTTP probe failed with statuscode: 503 ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances weaviate-yaghht --namespace ns-rruor ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME weaviate-yaghht-weaviate-0 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:25 UTC+0800 weaviate-yaghht-weaviate-1 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-12-213.us-west-2.compute.internal/172.31.12.213 Jun 19,2025 18:24 UTC+0800 check pod status done cluster hscale check cluster status before ops cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster weaviate-yaghht yaml-------------------------------------- `kubectl get cluster weaviate-yaghht -o yaml --namespace ns-rruor ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T11:06:06.191447213Z" creationTimestamp: "2025-06-19T10:15:18Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 5 labels: app.kubernetes.io/instance: weaviate-yaghht clusterdefinition.kubeblocks.io/name: weaviate clusterversion.kubeblocks.io/name: weaviate-1.18.0 name: weaviate-yaghht namespace: ns-rruor resourceVersion: "62198" uid: 9bfe8f1a-796c-4834-bda8-06064842be07 spec: affinity: podAntiAffinity: Preferred tenancy: SharedNode clusterDefinitionRef: weaviate clusterVersionRef: weaviate-1.18.0 componentSpecs: - componentDefRef: weaviate disableExporter: false name: weaviate replicas: 2 resources: limits: cpu: 100m memory: 512Mi requests: cpu: 100m memory: 512Mi serviceAccountName: kb-weaviate-yaghht volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: DoNotTerminate status: clusterDefGeneration: 2 components: weaviate: phase: Updating podsReady: false podsReadyTime: "2025-06-19T10:22:01Z" conditions: - lastTransitionTime: "2025-06-19T10:15:18Z" message: 'The operator has started the provisioning of Cluster: weaviate-yaghht' observedGeneration: 5 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:15:18Z" message: Successfully applied for resources observedGeneration: 5 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 5 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster weaviate-yaghht-------------------------------------- `kubectl describe cluster weaviate-yaghht --namespace ns-rruor ` Name: weaviate-yaghht Namespace: ns-rruor Labels: app.kubernetes.io/instance=weaviate-yaghht clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 Annotations: kubeblocks.io/reconcile: 2025-06-19T11:06:06.191447213Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:18Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 5 Resource Version: 62198 UID: 9bfe8f1a-796c-4834-bda8-06064842be07 Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: weaviate Cluster Version Ref: weaviate-1.18.0 Component Specs: Component Def Ref: weaviate Disable Exporter: false Name: weaviate Replicas: 2 Resources: Limits: Cpu: 100m Memory: 512Mi Requests: Cpu: 100m Memory: 512Mi Service Account Name: kb-weaviate-yaghht Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 2Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: DoNotTerminate Status: Cluster Def Generation: 2 Components: Weaviate: Phase: Updating Pods Ready: false Pods Ready Time: 2025-06-19T10:22:01Z Conditions: Last Transition Time: 2025-06-19T10:15:18Z Message: The operator has started the provisioning of Cluster: weaviate-yaghht Observed Generation: 5 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:15:18Z Message: Successfully applied for resources Observed Generation: 5 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T10:23:37Z Message: pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T10:23:37Z Message: pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 5 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 52m cluster-controller component is Creating Normal Running 48m (x2 over 50m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 48m (x2 over 50m) cluster-controller component is Running Normal AllReplicasReady 48m (x2 over 50m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 48m (x2 over 50m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 48m cluster-controller component is Stopping Normal Stopped 47m cluster-controller Cluster: weaviate-yaghht stopped successfully. Normal ComponentPhaseTransition 47m cluster-controller component is Stopped Normal ComponentPhaseTransition 46m (x2 over 49m) cluster-controller component is Updating Normal HorizontalScale 46m component-controller start horizontal scale component weaviate of cluster weaviate-yaghht from 0 to 2 Warning ReplicasNotReady 46m (x2 over 49m) cluster-controller pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Warning ComponentsNotReady 46m (x2 over 49m) cluster-controller pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Warning Unhealthy 42m (x3 over 42m) event-controller Pod weaviate-yaghht-weaviate-0: Readiness probe failed: HTTP probe failed with statuscode: 503 Normal ApplyResourcesSucceed 26m (x5 over 52m) cluster-controller Successfully applied for resources Normal PreCheckSucceed 26m (x5 over 52m) cluster-controller The operator has started the provisioning of Cluster: weaviate-yaghht Warning Unhealthy 96s (x28 over 40m) event-controller Pod weaviate-yaghht-weaviate-1: Readiness probe failed: HTTP probe failed with statuscode: 503 ------------------------------------------------------------------------------------------------------------------ cluster update terminationPolicy WipeOut `kbcli cluster update weaviate-yaghht --termination-policy=WipeOut --namespace ns-rruor ` cluster.apps.kubeblocks.io/weaviate-yaghht updated check cluster status `kbcli cluster list weaviate-yaghht --show-labels --namespace ns-rruor ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS weaviate-yaghht ns-rruor weaviate weaviate-1.18.0 WipeOut Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=weaviate-yaghht,clusterdefinition.kubeblocks.io/name=weaviate,clusterversion.kubeblocks.io/name=weaviate-1.18.0 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster weaviate-yaghht yaml-------------------------------------- `kubectl get cluster weaviate-yaghht -o yaml --namespace ns-rruor ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T11:16:05.196547432Z" creationTimestamp: "2025-06-19T10:15:18Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 6 labels: app.kubernetes.io/instance: weaviate-yaghht clusterdefinition.kubeblocks.io/name: weaviate clusterversion.kubeblocks.io/name: weaviate-1.18.0 name: weaviate-yaghht namespace: ns-rruor resourceVersion: "69434" uid: 9bfe8f1a-796c-4834-bda8-06064842be07 spec: affinity: podAntiAffinity: Preferred tenancy: SharedNode clusterDefinitionRef: weaviate clusterVersionRef: weaviate-1.18.0 componentSpecs: - componentDefRef: weaviate disableExporter: false name: weaviate replicas: 2 resources: limits: cpu: 100m memory: 512Mi requests: cpu: 100m memory: 512Mi serviceAccountName: kb-weaviate-yaghht volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: WipeOut status: clusterDefGeneration: 2 components: weaviate: phase: Updating podsReady: false podsReadyTime: "2025-06-19T10:22:01Z" conditions: - lastTransitionTime: "2025-06-19T10:15:18Z" message: 'The operator has started the provisioning of Cluster: weaviate-yaghht' observedGeneration: 6 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:15:18Z" message: Successfully applied for resources observedGeneration: 6 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T10:23:37Z" message: 'pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 6 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster weaviate-yaghht-------------------------------------- `kubectl describe cluster weaviate-yaghht --namespace ns-rruor ` Name: weaviate-yaghht Namespace: ns-rruor Labels: app.kubernetes.io/instance=weaviate-yaghht clusterdefinition.kubeblocks.io/name=weaviate clusterversion.kubeblocks.io/name=weaviate-1.18.0 Annotations: kubeblocks.io/reconcile: 2025-06-19T11:16:05.196547432Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:15:18Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 6 Resource Version: 69434 UID: 9bfe8f1a-796c-4834-bda8-06064842be07 Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: weaviate Cluster Version Ref: weaviate-1.18.0 Component Specs: Component Def Ref: weaviate Disable Exporter: false Name: weaviate Replicas: 2 Resources: Limits: Cpu: 100m Memory: 512Mi Requests: Cpu: 100m Memory: 512Mi Service Account Name: kb-weaviate-yaghht Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 2Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: WipeOut Status: Cluster Def Generation: 2 Components: Weaviate: Phase: Updating Pods Ready: false Pods Ready Time: 2025-06-19T10:22:01Z Conditions: Last Transition Time: 2025-06-19T10:15:18Z Message: The operator has started the provisioning of Cluster: weaviate-yaghht Observed Generation: 6 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:15:18Z Message: Successfully applied for resources Observed Generation: 6 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T10:23:37Z Message: pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T10:23:37Z Message: pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 6 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ClusterReady 57m (x2 over 59m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal Running 57m (x2 over 59m) cluster-controller Cluster: weaviate-yaghht is ready, current phase is Running Normal ComponentPhaseTransition 57m (x2 over 59m) cluster-controller component is Running Normal AllReplicasReady 57m (x2 over 59m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ComponentPhaseTransition 57m cluster-controller component is Stopping Normal ComponentPhaseTransition 56m cluster-controller component is Stopped Normal Stopped 56m cluster-controller Cluster: weaviate-yaghht stopped successfully. Normal ComponentPhaseTransition 55m (x2 over 58m) cluster-controller component is Updating Normal HorizontalScale 55m component-controller start horizontal scale component weaviate of cluster weaviate-yaghht from 0 to 2 Warning ReplicasNotReady 55m (x2 over 58m) cluster-controller pods are not ready in Components: [weaviate], refer to related component message in Cluster.status.components Warning ComponentsNotReady 55m (x2 over 58m) cluster-controller pods are unavailable in Components: [weaviate], refer to related component message in Cluster.status.components Warning Unhealthy 51m (x3 over 51m) event-controller Pod weaviate-yaghht-weaviate-0: Readiness probe failed: HTTP probe failed with statuscode: 503 Normal ApplyResourcesSucceed 8m45s (x6 over 61m) cluster-controller Successfully applied for resources Normal PreCheckSucceed 8m45s (x6 over 61m) cluster-controller The operator has started the provisioning of Cluster: weaviate-yaghht Warning Unhealthy 25s (x30 over 49m) event-controller Pod weaviate-yaghht-weaviate-1: Readiness probe failed: HTTP probe failed with statuscode: 503 ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances weaviate-yaghht --namespace ns-rruor ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME weaviate-yaghht-weaviate-0 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:25 UTC+0800 weaviate-yaghht-weaviate-1 ns-rruor weaviate-yaghht weaviate Running us-west-2a 100m / 100m 512Mi / 512Mi data:2Gi ip-172-31-12-213.us-west-2.compute.internal/172.31.12.213 Jun 19,2025 18:24 UTC+0800 check pod status done cluster list-logs `kbcli cluster list-logs weaviate-yaghht --namespace ns-rruor ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update weaviate-yaghht --enable-all-logs=true --namespace ns-rruor Error from server (NotFound): pods "weaviate-yaghht-weaviate-0" not found cluster logs `kbcli cluster logs weaviate-yaghht --tail 30 --namespace ns-rruor ` ***"action":"config_load","config_file_path":"/weaviate-config/conf.yaml","level":"info","msg":"Usage of the weaviate.conf.json file is deprecated and will be removed in the future. Please use environment variables.","time":"2025-06-19T10:26:15Z"*** ***"deprecation":***"apiType":"Configuration","id":"config-files","locations":["--config-file=\"\""],"mitigation":"Configure Weaviate using environment variables.","msg":"use of deprecated command line argument --config-file","sinceTime":"2020-09-08T09:46:00.000Z","sinceVersion":"0.22.16","status":"deprecated"***,"level":"warning","msg":"use of deprecated command line argument --config-file","time":"2025-06-19T10:26:15Z"*** ***"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2025-06-19T10:26:15Z"*** ***"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2025-06-19T10:26:15Z"*** ***"action":"lsm_recover_from_active_wal","class":"Questionznzrt","index":"questionznzrt","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/questionznzrt_EgqAxzNKqLeH_lsm/objects/segment-1750328611787135908.wal","shard":"EgqAxzNKqLeH","time":"2025-06-19T10:26:15Z"*** ***"action":"lsm_recover_from_active_wal_success","class":"Questionznzrt","index":"questionznzrt","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/questionznzrt_EgqAxzNKqLeH_lsm/objects/segment-1750328611787135908.wal","shard":"EgqAxzNKqLeH","time":"2025-06-19T10:26:15Z"*** ***"action":"lsm_recover_from_active_wal","class":"Questionznzrt","index":"questionznzrt","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/questionznzrt_EgqAxzNKqLeH_lsm/property__id/segment-1750328611791673022.wal","shard":"EgqAxzNKqLeH","time":"2025-06-19T10:26:15Z"*** ***"action":"lsm_recover_from_active_wal_success","class":"Questionznzrt","index":"questionznzrt","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/questionznzrt_EgqAxzNKqLeH_lsm/property__id/segment-1750328611791673022.wal","shard":"EgqAxzNKqLeH","time":"2025-06-19T10:26:15Z"*** ***"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"questionznzrt_EgqAxzNKqLeH","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2025-06-19T10:26:15Z","took":674587*** ***"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2025-06-19T10:26:15Z"*** ***"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2025-06-19T10:26:16Z"*** delete cluster weaviate-yaghht `kbcli cluster delete weaviate-yaghht --auto-approve --namespace ns-rruor ` Cluster weaviate-yaghht deleted pod_info:weaviate-yaghht-weaviate-0 1/1 Running 0 50m weaviate-yaghht-weaviate-1 0/1 Running 0 52m pod_info:weaviate-yaghht-weaviate-0 1/1 Terminating 0 51m weaviate-yaghht-weaviate-1 0/1 Terminating 0 52m No resources found in ns-rruor namespace. delete cluster pod done No resources found in ns-rruor namespace. check cluster resource non-exist OK: pvc No resources found in ns-rruor namespace. delete cluster done No resources found in ns-rruor namespace. No resources found in ns-rruor namespace. No resources found in ns-rruor namespace. Weaviate Test Suite All Done! --------------------------------------Weaviate (Topology = Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=weaviate;ClusterVersion=weaviate-1.18.0;]|[Description=Create a cluster with the specified cluster definition weaviate and cluster version weaviate-1.18.0] [PASSED]|[VolumeExpansion]|[ComponentName=weaviate]|[Description=VolumeExpansion the cluster specify component weaviate] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Connect]|[Endpoints=true]|[Description=Connect to the cluster] [FAILED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [FAILED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [FAILED]|[HorizontalScaling Out]|[ComponentName=weaviate]|[Description=HorizontalScaling Out the cluster specify component weaviate] [FAILED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]