Hello Dagster team I am using the k8sRunLauncher w out Celer

Question

Hello Dagster team I am using the k8sRunLauncher w out Celery It works great when I just have a single node that matches the nodeSelector However if I have more than one node it will also start the dagster jobs on those nodes but it fails to pull the docker image Seems that the imagePullSecrets are only getting loaded by the first node Any ideas Is this a Dagster issue or am I doing somthing wrong Error from failed job pod ```Events Type Reason Age From Message Normal Scheduled 69s default scheduler Successfully assigned dagster dagster run db90fb2b df8a 4bb2 af55 4cedecb91cb7 nc55c to lke10936 18246 5fd2d5d8e086 Normal BackOff 36s x2 over 67s kubelet Back off pulling image Warning Failed 36s x2 over 67s kubelet Error ImagePullBackOff Normal Pulling 24s x3 over 68s kubelet Pulling image Warning Failed 24s x3 over 68s kubelet Failed to pull image rpc error code = Unknown desc = Error response from daemon Get no basic auth credentials Warning Failed 24s x3 over 68s kubelet Error ErrImagePull``` My Helm values yaml file ```dagit nodeSelector 18034 postgresql master nodeSelector 18034 k8sRunLauncher enabled true env secrets aws secrets env nodeSelector 18386 userDeployments enabled true deployments name k8s user code image repository tag latest pullPolicy Always dagsterApiGrpcArgs f training repo py replicaCount 1 port 3030 env ENV VAR env config maps env secrets aws secrets env nodeSelector 18386 affinity tolerations podSecurityContext securityContext resources imagePullSecrets name dockerconfigjson github com celery enabled false rabbitmq enabled false```

Noah K · Answer

What does the YAML for the failing pod look like

Jeremy H · Answer

i ll need to repro and paste that here assume you mean the yaml that i get from ```kubectl get pod lt pod name gt o yaml```

Noah K · Answer

Yes

Noah K · Answer

Possibly with a ` n lt namespace gt ` if needed

Jeremy H · Answer

```apiVersion v1 kind Pod metadata annotations 10 2 40 2 32 creationTimestamp 2020 12 14T03 56 42Z generateName dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 labels run coordinator dagster dagster dagster 0 9 21 controller uid 1c2fafc8 e299 4d8e afda 87cb6be44e48 job name dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 managedFields apiVersion v1 fieldsType FieldsV1 fieldsV1 f metadata f generateName f labels f f f f f f controller uid f job name f ownerReferences k uid 1c2fafc8 e299 4d8e afda 87cb6be44e48 f apiVersion f blockOwnerDeletion f controller f kind f name f uid f spec f containers k name dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 f args f command f env k name DAGSTER HOME f name f value k name DAGSTER PG PASSWORD f name f valueFrom f secretKeyRef f key f name f envFrom f image f imagePullPolicy f name f resources f requests f cpu f terminationMessagePath f terminationMessagePolicy f volumeMounts k mountPath opt dagster dagster home dagster yaml f mountPath f name f subPath f dnsPolicy f enableServiceLinks f restartPolicy f schedulerName f securityContext f serviceAccount f serviceAccountName f terminationGracePeriodSeconds f volumes k name dagster instance f configMap f defaultMode f name f name manager kube controller manager operation Update time 2020 12 14T03 56 42Z apiVersion v1 fieldsType FieldsV1 fieldsV1 f metadata f annotations f manager calico operation Update time 2020 12 14T03 56 43Z apiVersion v1 fieldsType FieldsV1 fieldsV1 f status f conditions k type ContainersReady f lastProbeTime f lastTransitionTime f message f reason f status f type k type Initialized f lastProbeTime f lastTransitionTime f status f type k type Ready f lastProbeTime f lastTransitionTime f message f reason f status f type f containerStatuses f hostIP f podIP f podIPs k ip 10 2 40 2 f ip f startTime manager kubelet operation Update time 2020 12 14T03 56 45Z name dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 pb6gp namespace dagster ownerReferences apiVersion batch v1 blockOwnerDeletion true controller true kind Job name dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 uid 1c2fafc8 e299 4d8e afda 87cb6be44e48 resourceVersion 2443601 selfLink api v1 namespaces dagster pods dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 pb6gp uid 8ecf7420 8a72 48fc 9e28 33045a3b2c84 spec containers args api execute run with structured logs class ExecuteRunArgs instance ref null pipeline origin class PipelinePythonOrigin pipeline name training pipeline repository origin class RepositoryPythonOrigin code pointer class FileCodePointer fn name training repository python file training repo py working directory executable path usr local bin python pipeline run id 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 command dagster env name DAGSTER HOME value opt dagster dagster home name DAGSTER PG PASSWORD valueFrom secretKeyRef key postgresql password name dagster postgresql secret envFrom configMapRef name dagster pipeline env image imagePullPolicy IfNotPresent name dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 resources requests cpu 5 terminationMessagePath dev termination log terminationMessagePolicy File volumeMounts mountPath opt dagster dagster home dagster yaml name dagster instance subPath dagster yaml mountPath var run secrets kubernetes io serviceaccount name dagster token spqmw readOnly true dnsPolicy ClusterFirst enableServiceLinks true nodeName lke10936 18441 5fd6e111cdc5 priority 0 restartPolicy Never schedulerName default scheduler securityContext serviceAccount dagster serviceAccountName dagster terminationGracePeriodSeconds 30 tolerations effect NoExecute key operator Exists tolerationSeconds 300 effect NoExecute key operator Exists tolerationSeconds 300 volumes configMap defaultMode 420 name dagster instance name dagster instance name dagster token spqmw secret defaultMode 420 secretName dagster token spqmw status conditions lastProbeTime null lastTransitionTime 2020 12 14T03 56 42Z status True type Initialized lastProbeTime null lastTransitionTime 2020 12 14T03 56 42Z message containers with unready status dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 reason ContainersNotReady status False type Ready lastProbeTime null lastTransitionTime 2020 12 14T03 56 42Z message containers with unready status dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 reason ContainersNotReady status False type ContainersReady lastProbeTime null lastTransitionTime 2020 12 14T03 56 42Z status True type PodScheduled containerStatuses image imageID lastState name dagster run 7e0c0df5 c83e 4b8f 9d5c 4aec7f32dcc2 ready false restartCount 0 started false state waiting message Back off pulling image reason ImagePullBackOff hostIP 192 168 128 163 phase Pending podIP 10 2 40 2 podIPs ip 10 2 40 2 qosClass Burstable startTime 2020 12 14T03 56 42Z ```

Noah K · Answer

That doesn t appear to have any pull secrets on it

Noah K · Answer

Are you sure the helm chart values are set correctly

Jeremy H · Answer

strangely the jobs that do work also don t have the pull secrets in their yaml

Jeremy H · Answer

i am applying the values yaml file to the standard dagster chart using `helm upgrade dagster dagster dagster namespace dagster create namespace f infra helm dagster values yaml`

Jeremy H · Answer

is that what you mean by chart values are set correctly

Noah K · Answer

No I mean you showed what values are you using but are you sure those are correct slightly smiling face

Noah K · Answer

Can you link whatever chart you re using

Jeremy H · Answer

The name of the imagePullSecret and docker image are correct Not at all sure the rest is correct except that it works on one node

Noah K · Answer

Again not the data

Noah K · Answer

But the values structure

Noah K · Answer

You might have put things in the wrong place

Noah K · Answer

And the one node that is working may have global docker creds installed

Jeremy H · Answer

very possible that i have the structure wrong

Noah K · Answer

Can you please link to whatever Helm char you are using

Jeremy H · Answer

i installed using ```helm repo add dagster <https dagster io github io helm> helm install dagster dagster dagster```

Jeremy H · Answer

in the helm chart all the references to imagePullSecrets look like this `$ Values imagePullSecrets` so it seems I have those defined in the right place

Jeremy H · Answer

i don t have any global docker creds installed anywhere but the jobs that do work are on the same node as the k8sRunLauncher pod the one running the grpc api server and that pod does have the imagePullSecrets in its yaml so that might be related but i don t think that different pods running on the same node would see each other s imagePullSecre

Noah K · Answer

Looks about like I would expect dump the configmap containing the dagit config file that should show you the options being passed to the launcher

Jeremy H · Answer

this one ```$ kubectl get configmap dagster instance n dagster o yaml apiVersion v1 data dagster yaml scheduler n module dagster cron cron scheduler n class SystemCronScheduler n nschedule storage n module dagster postgres schedule storage n class PostgresScheduleStorage n config n postgres db n username test n password n env DAGSTER PG PASSWORD n hostname dagster postgresql n db name test n port 5432 n nrun launcher n module dagster k8s n class K8sRunLauncher n config n load incluster config true n kubeconfig file n job namespace dagster n service account name dagster n dagster home n env DAGSTER HOME n instance config map n env DAGSTER K8S INSTANCE CONFIG MAP n postgres password secret n env DAGSTER K8S PG PASSWORD SECRET n env config maps n env DAGSTER K8S PIPELINE RUN ENV CONFIGMAP n env secrets n nrun storage n module dagster postgres run storage n class PostgresRunStorage n config n postgres db n username test n password n env DAGSTER PG PASSWORD n hostname dagster postgresql n db name test n port 5432 n nevent log storage n module dagster postgres event log n class PostgresEventLogStorage n config n postgres db n username test n password n env DAGSTER PG PASSWORD n hostname dagster postgresql n db name test n port 5432 n kind ConfigMap metadata annotations dagster dagster creationTimestamp 2020 12 07T00 14 10Z labels app dagster Helm chart dagster 0 9 21 heritage Helm release dagster managedFields apiVersion v1 fieldsType FieldsV1 fieldsV1 f data f dagster yaml f metadata f annotations f f f labels f app f f chart f heritage f release manager Go http client operation Update time 2020 12 07T21 41 03Z name dagster instance namespace dagster resourceVersion 2155420 selfLink api v1 namespaces dagster configmaps dagster instance uid 167d0873 fc5d 474e 9e04 f12fe3003680```

Noah K · Answer

Bit hard to read through all that formatting but I don t think I see an image pull secrets on the launcher config

Jeremy H · Answer

correct do i need to set the pull secrets property in the config i was expecting it to get set from the helm side

Jeremy H · Answer

but i can add it to the config if that s the right place for it

Noah K · Answer

Helm doesn t create the job pods Dagit does

Noah K · Answer

But Helm should be configuring Dagit

Noah K · Answer

That s what that configmap does

Noah K · Answer

Noah K · Answer

That looks correct

Noah K · Answer

And I think that matches your values

Noah K · Answer

The `$ Values` is weird but should work

Jeremy H · Answer

so you think the values from helm should land in the configmap but they are not

Noah K · Answer

Oh wait that s the scheduler

Noah K · Answer

Yeah if I scroll down to the right section

Noah K · Answer

This is a bug in the helm chart

Noah K · Answer

it doesn t pass through the config correctly

Noah K · Answer

Noah K · Answer

Nothing in there sets image pull secrets

Jeremy H · Answer

yup

Noah K · Answer

But it is supported in the launcher <https github com dagster io dagster blob a07fc28ae6878235aeada88e3584c6dea01f845b python modules libraries dagster k8s dagster k8s launcher py L80 L87>

Jeremy H · Answer

any idea why it works on that one node but not others does it get the secrets somehow from the other pod on that same node

Noah K · Answer

Because the image is already pulled there from other deployments which do have the pull secret

Noah K · Answer

So the kubelet doesn t try to pull it again

Noah K · Answer

However on a node which doesn t have it already it tries and fails

Jeremy H · Answer

oh right so the image is already cached on that node

Jeremy H · Answer

got it

Jeremy H · Answer

ok i ll figure out how to get that value into the launcher config

Noah K · Answer

You cannot without forking and fixing the helm chart

Noah K · Answer

I mean you could hand edit the configmap but that will put it out of sync with Helm so I wouldn t recommend it

Jeremy H · Answer

is there a way to set it from the python side

Jeremy H · Answer

i guess for now i can also run another dummy pod on that node to pull the image

Noah K · Answer

Or just fix the chart slightly smiling face

Noah K · Answer

It s a 3 line change

alex · Answer

< rex>