Hi All i need help on k8s i have set on the job this decorat

Question

Hi All i need help on k8s i have set on the job this decoration ``` tags= dagster k8s config container config resources requests cpu 4000m memory 8000Mi limits cpu 4000m memory 16000Mi pod spec config affinity nodeAffinity requiredDuringSchedulingIgnoredDuringExecution nodeSelectorTerms matchExpressions key operator In values immunai pipeline pool ``` and when we run a jobs the pod creation is stuck ``` Warning FailedScheduling 41m default scheduler 0 4 nodes are available 1 Insufficient cpu 1 Insufficient memory 1 node s were unschedulable 2 node s didn t match Pod s node affinity selector Warning FailedScheduling 41m default scheduler 0 4 nodes are available 1 Insufficient cpu 1 Insufficient memory 1 node s were unschedulable 2 node s didn t match Pod s node affinity selector Normal NotTriggerScaleUp 36s x241 over 41m cluster autoscaler pod didn t trigger scale up 1 node s didn t match Pod s node affinity selector``` in the nodes the label exist ```gke dagster omic dat immunai pipeline ca9c3a0a 60cw Ready lt none gt 2d21h v1 21 6 gke 1500 ```

Andrea Giardini · Answer

Can you post the full description of the pod

Igal Dahan · Answer

yes

Igal Dahan · Answer

```kubectl describe po dagster run 4cc9fae2 0ae5 45b3 9862 11278066c628 vhm6m Name dagster run 4cc9fae2 0ae5 45b3 9862 11278066c628 vhm6m Namespace default Priority 0 Node lt none gt Labels controller uid=cee1d7f5 67ae 416d 8ead 9e2f5d6f1b77 job name=dagster run 4cc9fae2 0ae5 45b3 9862 11278066c628 Annotations lt none gt Status Pending IP IPs lt none gt Controlled By Job dagster run 4cc9fae2 0ae5 45b3 9862 11278066c628 Containers dagster Image Port lt none gt Host Port lt none gt Args usr bin python3 m dagster api execute run class ExecuteRunArgs instance ref null pipeline origin class PipelinePythonOrigin pipeline name staging single sample job repository origin class RepositoryPythonOrigin code pointer class ModuleCodePointer fn name staging single sample repo module single sample pipeline container image executable path usr bin python3 pipeline run id 4cc9fae2 0ae5 45b3 9862 11278066c628 Limits cpu 4 memory 16000Mi Requests cpu 4 memory 8000Mi Environment Variables from dagster pipeline env ConfigMap Optional false Environment DAGSTER HOME opt dagster dagster home DAGSTER PG PASSWORD lt set to the key postgresql password in secret dagster postgresql secret gt Optional false LD LIBRARY PATH Mounts opt dagster dagster home dagster yaml from dagster instance rw path= dagster yaml var run secrets kubernetes io serviceaccount from kube api access j2chw ro Conditions Type Status PodScheduled False Volumes dagster instance Type ConfigMap a volume populated by a ConfigMap Name dagster instance Optional false kube api access j2chw Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class Burstable Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events Type Reason Age From Message Warning FailedScheduling 41m default scheduler 0 4 nodes are available 1 Insufficient cpu 1 Insufficient memory 1 node s were unschedulable 2 node s didn t match Pod s node affinity selector Warning FailedScheduling 41m default scheduler 0 4 nodes are available 1 Insufficient cpu 1 Insufficient memory 1 node s were unschedulable 2 node s didn t match Pod s node affinity selector Normal NotTriggerScaleUp 36s x241 over 41m cluster autoscaler pod didn t trigger scale up 1 node s didn t match Pod s node affinity selector```

Andrea Giardini · Answer

I don t see the affinity in the pod description

Andrea Giardini · Answer

I think you need to replace the camel case with the underscore syntax

Andrea Giardini · Answer

nodeAffinity gt node affinity

Andrea Giardini · Answer

```requiredDuringSchedulingIgnoredDuringExecution gt required during scheduling ignored during execution``` etc etc

Igal Dahan · Answer

hmm

Igal Dahan · Answer

is it documented

Igal Dahan · Answer

and what is it camel case is it a phrase

Andrea Giardini · Answer

gt is it documented <https docs dagster io deployment guides kubernetes customizing your deployment job or op kubernetes configuration> I see the docs here say exactly what you do so I might be wrong

Andrea Giardini · Answer

gt and what is it camel case is it a phrase + <https en wikipedia org wiki Camel case>

Igal Dahan · Answer

when we did edit pod we see ```spec affinity nodeAffinity requiredDuringSchedulingIgnoredDuringExecution nodeSelectorTerms matchExpressions key <http cloud google com gke nodepool|cloud google com gke nodepool> operator In values immunai pipeline pool```

Andrea Giardini · Answer

Weird if the affinity is there and the node label is there it should work

Andrea Giardini · Answer

what about `kubectl get pod $podname o yaml`

Igal Dahan · Answer

i have also another guy in the team that request to join and he is still in pending could you approve him his email <mailto eldan hamdani immunai com|eldan hamdani immunai com>

Andrea Giardini · Answer

I am not sure what you are referring to

Igal Dahan · Answer

```apiVersion v1 kind Pod metadata creationTimestamp 2022 02 10T10 55 56Z generateName dagster run e47b5c8a 09a6 4052 bef9 d0e9f6cfab6f labels run worker dagster dagster dagster 0 13 4 controller uid 4021f273 5ee1 432b a31e 3d4e91197860 job name dagster run e47b5c8a 09a6 4052 bef9 d0e9f6cfab6f name dagster run e47b5c8a 09a6 4052 bef9 d0e9f6cfab6f zpwh5 namespace default ownerReferences apiVersion batch v1 blockOwnerDeletion true controller true kind Job name dagster run e47b5c8a 09a6 4052 bef9 d0e9f6cfab6f uid 4021f273 5ee1 432b a31e 3d4e91197860 resourceVersion 14284445 uid 1fdf8d27 a718 490c a521 d11a1597367c spec affinity nodeAffinity requiredDuringSchedulingIgnoredDuringExecution nodeSelectorTerms matchExpressions key operator In values immunai pipeline pool containers args usr bin python3 m dagster api execute run class ExecuteRunArgs instance ref null pipeline origin class PipelinePythonOrigin pipeline name staging single sample job repository origin class RepositoryPythonOrigin code pointer class ModuleCodePointer fn name staging single sample repo module single sample pipeline container image executable path usr bin python3 pipeline run id e47b5c8a 09a6 4052 bef9 d0e9f6cfab6f env name DAGSTER HOME value opt dagster dagster home name DAGSTER PG PASSWORD valueFrom secretKeyRef key postgresql password name dagster postgresql secret name LD LIBRARY PATH envFrom configMapRef name dagster pipeline env image imagePullPolicy Always name dagster resources limits cpu 4 memory 16000Mi requests cpu 4 memory 8000Mi terminationMessagePath dev termination log terminationMessagePolicy File volumeMounts mountPath opt dagster dagster home dagster yaml name dagster instance subPath dagster yaml mountPath var run secrets kubernetes io serviceaccount name kube api access wch7g readOnly true dnsPolicy ClusterFirst enableServiceLinks true imagePullSecrets name gcr json key preemptionPolicy PreemptLowerPriority priority 0 restartPolicy Never schedulerName default scheduler securityContext serviceAccount dagster serviceAccountName dagster terminationGracePeriodSeconds 30 tolerations effect NoExecute key operator Exists tolerationSeconds 300 effect NoExecute key operator Exists tolerationSeconds 300 volumes configMap defaultMode 420 name dagster instance name dagster instance name kube api access wch7g projected defaultMode 420 sources serviceAccountToken expirationSeconds 3607 path token configMap items key ca crt path ca crt name kube root ca crt downwardAPI items fieldRef apiVersion v1 fieldPath metadata namespace path namespace status conditions lastProbeTime null lastTransitionTime 2022 02 10T10 55 56Z message 0 4 nodes are available 1 Insufficient cpu 1 Insufficient memory 1 node s were unschedulable 2 node s didn t match Pod s node affinity selector reason Unschedulable status False type PodScheduled phase Pending qosClass Burstable```

Igal Dahan · Answer

a guy the want to join this slack channel

Igal Dahan · Answer

i was not clear sorry

Andrea Giardini · Answer

I have no control over the slack channel I am just a member of the community The slack server and channel are free to join

Igal Dahan · Answer

i see now its working for use the resources should fit

Igal Dahan · Answer

thanks for the help

Igal Dahan · Answer

champagne

Andrea Giardini · Answer

Good to know let me know in case you need additional help slightly smiling face I ve been working with Dagster for some time and I am looking for more projects with it