Hey all wave I m new to both Dagster and Kubernetes but real

Question

Hey all wave I m new to both Dagster and Kubernetes but really excited for what looks like an awesome tool I ve been trying to deploy Dagster on AWS EKS following the <https docs dagster io deploying k8s|Deploying on Kubernetes> guide but have been running into issues getting the `dagster daemon` and `dagster k8s example user code 1` pods to deploy successfully `dagster dagit` works I followed the Default Deployment section with the only exception that I launched a Postgres RDS instance and updated the credentials for Postgres in `values yaml` Yet those 2 pods keep erroring with the following message ```Startup probe errored rpc error code = DeadlineExceeded desc = failed to exec in container timeout 3s exceeded context deadline exceeded``` I m wondering if there s any chance this is related to Dagster s Helm or if anyone here has encountered this issue trying to deploy this example Otherwise if this sounds entirely related to Kubernetes I m happy to keep searching Thanks pray

Charles Lariviere · Answer

Here s the event logs from those 2 pods ```Type Reason Age From Message Warning LoggingDisabled 7m21s fargate scheduler Disabled logging because aws logging configmap was not found configmap aws logging not found Normal Scheduled 5m31s fargate scheduler Successfully assigned default dagster k8s example user code 1 77b57b6f5b rl8n5 to fargate ip 192 168 93 206 ec2 internal Normal Pulling 5m31s kubelet Pulling image postgres 9 6 16 Normal Pulled 5m24s kubelet Successfully pulled image postgres 9 6 16 Normal Created 5m21s kubelet Created container check db ready Normal Started 5m21s kubelet Started container check db ready Normal Pulling 5m20s kubelet Pulling image dagster k8s example latest Normal Pulled 4m54s kubelet Successfully pulled image dagster k8s example latest Normal Created 4m53s kubelet Created container dagster Normal Started 4m52s kubelet Started container dagster Warning Unhealthy 22s x26 over 4m32s kubelet Startup probe errored rpc error code = DeadlineExceeded desc = failed to exec in container timeout 3s exceeded context deadline exceeded```

mrdavidlaing · Answer

Are you able to exec into the daemon container and manually run the command being used by startup probe <https github com dagster io dagster blob master helm dagster values yaml L242|https github com dagster io dagster blob master helm dagster values yaml L242> I m wondering if that might give some more detailed error information

Jason · Answer

I ve just started looking at the k8s example as well and got it working with RDS postgres never got error messages I noticed in your errors it mentions `fargate` are you using EKS on Fargate Maybe that s part of the issue Try just the managed EKS as a test first `helm install dagster dagster dagster f values yaml set runLauncher type=K8sRunLauncher set userDeployment enabled=true` ```apiVersion <http eksctl io v1alpha5|eksctl io v1alpha5> kind ClusterConfig metadata name lt my cluster name gt region lt region gt managedNodeGroups name ng dagster instanceType m5 large minSize 2 desiredCapacity 2 maxSize 3 ssh allow true publicKeyName lt My KeyPair gt ```

Charles Lariviere · Answer

Moving to Managed nodes fixed it for me thanks pray