https://dagster.io/ logo
#ask-community
Title
# ask-community
e

Eduard Ethan Carrés Hidalgo

02/13/2023, 4:45 PM
Hi there! I have a question, regarding deployments in Kubernetes. As I've understood, DAGs live in code locations, which are in code location servers that use a certain docker image. That docker image HAS to be the same that's used for executing the code inherent to those tags, am I correct? We want to introduce the usage of Dagster into all the teams of a certain area of our company. Each team has it's own repository and docker image, and will use Dagster internally to parallelize several parts of it's code. But we would also want to have a centralized place to use Dagster to orchestrate those different images, because we run pipelines that execute all of them. Does it mean we need to generate a single, all-in-one docker image that's able to run any OP of the DAGs? I hope it's understandable, if not please let me know and I'll try to rephrase or do a graphic representation of the issue hat tip
k

Kirill Yakunin

02/13/2023, 5:20 PM
Hello! I am currently working on developing Dagster infrastructure, I believe we have very similar requirements. Luckily, you don't need to create a single super-image. You can create several executor-images, and then set image per-job or per-op (you can also set images per-asset, please see my question above): https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment It's important, that your executor-images have to use you user-deployment image as a base. And you must not redefine CMD or ENTRYPOINTS. But you can install any dependencies that you need to run your code. If you will have further questions - please contact me, I will try to help. It's a pity, that this stuff is not documented, had to figure out a lot by myself.
e

Eduard Ethan Carrés Hidalgo

02/14/2023, 1:51 PM
Wow,many thanks! Yes, I did not found any documentation, so I'm very thankful of your answer hat tip
k

Kirill Yakunin

02/14/2023, 2:20 PM
You are welcome! Update: your executor images don't have to use the user-code-deployment image as a base. They just need to have Dagster and libs (dagster-gcp, dagster-postgresql, ...) installed.
2 Views