https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
e

Eegan K

07/21/2022, 2:05 PM
Hey guys, been working with dagster a ton over the last few months. Y’all are killing it. Finally gotten to the point where I’m trying to deploy dagster on Kubernetes, and I was surprised to find your documentation lacking. I’m particularly frustrated by the lack of documentation on creating one's own code images. The user code directory still refers to “pipelines”, and the example docker file shows very little structure. The user-code-example image isn't even available if I wanted to use it. It is quite possible that I’ve misconfigured stuff with my Kubernetes config (I’m new to k8s), but from what I can see, it’s an issue with my code image. Some more info about my environment: • Host machine: Ubuntu 20.04, behind a corporate proxy • Using Minikube for testing (spent a while getting it working, but it works) Custom user code image hosted on a local, insecure registry • User code image’s entry point is dragster api grpc <package name>, <attribute name> <host> <port (3030)> • All images load fine, start to run fine (dagit, dagster-daemon, postgresql, etc) • User code image gives “CrashLoopBackOff” when I run kubectl get pods • When I use kubectl describe pods <user code image name>, it says something about “failed to pick subchannel”. I think this means that there’s an issue with grpc, but not sure if that’s on my side or the dagster helm side. Super thankful for any and all help! You guys are making something great here!
d

daniel

07/21/2022, 2:11 PM
Hi Eegan - thanks for the docs feedback and totally agree. The big thing that would be helpful is logs from that pod that is giving you a CrashLoopBackOff
kubectl logs <pod name>
usually there's a clue there as to why its crashing
e

Eegan K

07/21/2022, 2:15 PM
Thanks so much haha. Like I said I’m new to k8s. Looks like the problem (at the moment, at least) is with my own code. I’ll reply here if I have any more issues!
👍 1
s

Stephen Bailey

07/23/2022, 2:32 AM
the good news @Eegan K is that once you get it set up to your satisfaction, it requires not too much maintenance (in my experience). if you look back through this and other channels you'll see me shouting, but not so much anymore. 🙂 hang in there!
e

Eegan K

07/26/2022, 7:50 PM
Hey everyone! Finally fixed the issue on my own code’s side. Hopefully I have a bit more experience with k8s as well! I have a few questions now: 1. I tried changing the default Postgres credentials and database from “test” to something else (in the values.yml file). This caused the dagster-daemon pod to error because of invalid credentials, but I changed it back to “test” and now everything works. Any ideas? 2. What should the default entry point/command be for the user-supplied code image? Should it be an entry point or a command?
Again, thanks so much for your great work!
d

daniel

07/26/2022, 7:52 PM
Hi Eegan - dagster will inject the command for you. As long as the command 'dagster' works in the image, you should be fine. Re: postgres, you can configure the postgres config here in the helm chart: https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L665-L684 - not sure if this answers your question. I do see a comment there about credentials sticking around with a link to a github issue
👍 1
e

Eegan K

07/26/2022, 7:55 PM
For anyone in the future: this means that it SHOULD NOT be an entrypoint. More info here: https://codewithyury.com/docker-run-vs-cmd-vs-entrypoint/. Long story short, an entrypoint will always run, whereas cmd will just set a default which dagster will override.
2 Views