How I can apply node selector to all pods? #35679
-
|
I have many CPU and GPU nodes in my k3s cluster, I want to install milvus cluster with helm into fixed node using The problem is, I applied How I can do that? Here is my helm-milvus.yaml. rootCoordinator:
nodeSelector:
node-type: cpu
region: hangzhou
replicas: 1
resources:
limits:
cpu: 1
memory: 2Gi
queryCoordinator:
nodeSelector:
node-type: cpu
region: hangzhou
replicas: 1
resources:
limits:
cpu: "1"
memory: 1Gi
dataCoordinator:
nodeSelector:
node-type: cpu
region: hangzhou
replicas: 1
resources:
limits:
cpu: "1"
memory: 1Gi
proxy:
nodeSelector:
node-type: cpu
region: hangzhou
replicas: 1
resources:
limits:
cpu: 2
memory: 8Gi
queryNode:
nodeSelector:
node-type: cpu
region: hangzhou
replicas: 36
resources:
limits:
cpu: 16
memory: 128Gi
dataNode:
nodeSelector:
node-type: cpu
region: hangzhou
replicas: 1
resources:
limits:
cpu: 2
memory: 8Gi
indexNode:
nodeSelector:
node-type: cpu
region: hangzhou
replicas: 4
resources:
limits:
cpu: 8
memory: 16Gi
etcd:
nodeSelector:
node-type: cpu
region: hangzhou
autoCompactionMode: revision
autoCompactionRetention: "1000"
extraEnvVars:
- name: ETCD_QUOTA_BACKEND_BYTES
value: "4294967296"
- name: ETCD_HEARTBEAT_INTERVAL
value: "500"
- name: ETCD_ELECTION_TIMEOUT
value: "25000"
- name: ETCD_SNAPSHOT_COUNT
value: "10000"
- name: ETCD_ENABLE_PPROF
value: "true"
persistence:
accessMode: ReadWriteOnce
enabled: true
size: 30Gi #SSD Required
storageClass:
replicaCount: 3
resources:
limits:
cpu: 8
memory: 16Gi
requests:
cpu: 8
memory: 16Gi
pulsar:
nodeSelector:
node-type: cpu
region: hangzhou
enabled: true
recovery:
nodeSelector:
node-type: cpu
region: hangzhou
initContainers:
nodeSelector:
node-type: cpu
region: hangzhou
bookie:
nodeSelector:
node-type: cpu
region: hangzhou
initContainers:
nodeSelector:
node-type: cpu
region: hangzhou
proxy:
nodeSelector:
node-type: cpu
region: hangzhou
resources:
requests:
memory: 18Gi
cpu: 8
replicaCount: 2
configData:
PULSAR_MEM: >
-Xms8192M
-Xmx8192M
-XX:MaxDirectMemorySize=8192M
httpNumThreads: "100"
zookeeper:
nodeSelector:
node-type: cpu
region: hangzhou
volumes:
persistence: true
data:
name: data
size: 20Gi #SSD Required
storageClassName:
resources:
requests:
memory: 8Gi
cpu: 4
configData:
PULSAR_MEM: >
-Xms4096M
-Xmx4096M
bookkeeper:
nodeSelector:
node-type: cpu
region: hangzhou
volumes:
journal:
name: journal
size: 440Gi
storageClassName:
ledgers:
name: ledgers
size: 1100Gi #SSD Required
storageClassName:
resources:
requests:
memory: 64Gi
cpu: 8
configData:
PULSAR_MEM: >
-Xms16384M
-Xmx16384M
-XX:MaxDirectMemorySize=32768M
broker:
nodeSelector:
node-type: cpu
region: hangzhou
component: broker
podMonitor:
enabled: false
replicaCount: 2
resources:
requests:
memory: 72Gi
cpu: 8
configData:
PULSAR_MEM: >
-Xms16384M
-Xmx16384M
-XX:MaxDirectMemorySize=32768M
minio:
nodeSelector:
node-type: cpu
region: hangzhou
resources:
limits:
cpu: 8
memory: 32Gi
persistence:
storageClass:
accessMode: ReadWriteOnce
size: 6730GiHere is the result of $ kubectl describe pod milvus-pulsar-recovery-0 -n milvus
Name: milvus-pulsar-recovery-0
Namespace: milvus
Priority: 0
Service Account: default
Node: gpu-1/192.168.0.31
Start Time: Fri, 23 Aug 2024 14:55:23 +0800
Labels: app=pulsar
apps.kubernetes.io/pod-index=0
cluster=milvus-pulsar
component=recovery
controller-revision-hash=milvus-pulsar-recovery-65f7f8755b
release=milvus
statefulset.kubernetes.io/pod-name=milvus-pulsar-recovery-0
Annotations: prometheus.io/port: 8000
prometheus.io/scrape: true
Status: Running
IP: 10.42.1.24
IPs:
IP: 10.42.1.24
Controlled By: StatefulSet/milvus-pulsar-recovery
Init Containers:
pulsar-bookkeeper-verify-clusterid:
Container ID: containerd://eede809e4b645f9a0354ecffd068c84dc03eda0d2d75b803bc70de7474ce5f5c
Image: apachepulsar/pulsar:2.9.5
Image ID: docker.io/apachepulsar/pulsar@sha256:9bcbad78a65a09b3fe5d3acd158574ad0eda4531cc8c49db16760a2ed1364070
Port: <none>
Host Port: <none>
Command:
sh
-c
Args:
bin/apply-config-from-env.py conf/bookkeeper.conf;until bin/bookkeeper shell whatisinstanceid; do
sleep 3;
done;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 23 Aug 2024 14:55:24 +0800
Finished: Fri, 23 Aug 2024 14:55:26 +0800
Ready: True
Restart Count: 0
Environment Variables from:
milvus-pulsar-recovery ConfigMap Optional: false
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zf7sk (ro)
Containers:
milvus-pulsar-recovery:
Container ID: containerd://c54c5e72a74a76474995eeea07ccf8dfcda3f2108bb55bacbb5d55c0752784dd
Image: apachepulsar/pulsar:2.9.5
Image ID: docker.io/apachepulsar/pulsar@sha256:9bcbad78a65a09b3fe5d3acd158574ad0eda4531cc8c49db16760a2ed1364070
Port: 8000/TCP
Host Port: 0/TCP
Command:
sh
-c
Args:
bin/apply-config-from-env.py conf/bookkeeper.conf;
OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/bookkeeper autorecovery
State: Running
Started: Fri, 23 Aug 2024 14:55:26 +0800
Ready: True
Restart Count: 0
Requests:
cpu: 1
memory: 512Mi
Environment Variables from:
milvus-pulsar-recovery ConfigMap Optional: false
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zf7sk (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-zf7sk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none> |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 7 replies
-
|
/assign @LoveEachDay |
Beta Was this translation helpful? Give feedback.
-
|
@xiaofan-luan @LoveEachDay Hi, sorry to bother, is there any progress? |
Beta Was this translation helpful? Give feedback.
-
I belive those are one time pod and after pulsar start those pod will be terminated automatically. |
Beta Was this translation helpful? Give feedback.
-
|
@LucienShui We'll add a section to describe how to use |
Beta Was this translation helpful? Give feedback.
-
|
@LoveEachDay Thank you very much, I would really appreciate it if you could let me know when the documentation is ready. |
Beta Was this translation helpful? Give feedback.
-
Ok, I'll link the doc here. |
Beta Was this translation helpful? Give feedback.
-
|
@LucienShui https://github.com/zilliztech/milvus-helm/tree/master/charts/milvus#node-selector-affinity-and-tolerations-configuration-guide Here we add a guide for nodeSelector, affinity and toleration setup for milvus components and all dependencies. |
Beta Was this translation helpful? Give feedback.
@LucienShui https://github.com/zilliztech/milvus-helm/tree/master/charts/milvus#node-selector-affinity-and-tolerations-configuration-guide
Here we add a guide for nodeSelector, affinity and toleration setup for milvus components and all dependencies.