Hey everyone! I’m using both the DockerRunLauncher...
# ask-community
m
Hey everyone! I’m using both the DockerRunLauncher and the docker_executor. Right now there is only one docker image that is used to build containers that execute job steps. It would be great to have more than one image and to be able to specify which step uses which image. Is it currently possible with Dagster? Thanks! 🙏
🤖 1
j
Hi Meder, I’m curious what your use case is. Are you looking to run a dagster op inside a different image? Would you be running some other cli command?
m
Hi Johann! Yes. Some ops have dependency conflicts and running them in different images seemed like a solution. Additionally, and this doesn’t have to be done via docker, it would be awesome to be able to specify a worker for an op. Some ops are memory intensive, while others might require a GPU, and so it would be great to be able to specify that somehow.
j
Got it. Both of these features are things we have for K8s, but haven’t ported to Docker. Can put them on the queue
@Dagster Bot issue Allow per step image and container config with docker_executor
d
m
Thanks!
Hey @johann! I decided to give dagster_k8s a go. I can get the K8sRunLauncher working, but how do I specify workers per op? As mentioned above, I would like to be able to select a specific image for it, which in this case is a special worker. But from what I see K8sRunLauncher uses the same type of image for all workers.
j
To use a different image for an op, you’ll want to use the
k8s_job_executor
. Then you can specify an image in tags on each Op: https://dagster.slack.com/archives/C01U954MEER/p1644335565408589?thread_ts=1644334246.680049&cid=C01U954MEER
🙏 1