Hi Dagster team, I’m curious about workarounds for this github issue, where I want to use dagster to run several solids within the same pipeline using different docker images. For background, we use

CeleryK8sRunLauncher

which allows us to have resource/compute separation of solids within a pipeline, but seems like dagster doesn’t allow us to have package / environment separation natively.

Hi Liu - it'd take a very small change on our side for the container_config solution to work, we can go ahead and do that for the release this week. once we do that, setting the image in solid tags for container_config should work. You actually don't need to set the CMD in the image to anything in order for that to work, the dagster executor will override it to the right thing. However, the big restriction in order for this to work reliably right now, you need the pipeline code to be exactly the same across all the different images being used during a run (including the run worker image as well as any images that you override for individual solids), and the pipeline needs to be located at exactly the same place in each image. If you were to update the pipeline code being used for one solid's image without updating the code for the rest of them, you might get really confusing errors.

Hi Daniel, that should be interesting for us to test! I imagine what we could do is simply copy over the dagster pipeline artifacts to each docker image. For different images of different solids, does the python environment also need to live in the same path across different images?

I don't think the python environment needs to have the same path, no - the important thing is that all the different images be able to load the exact same PipelineDefinition.

I see… Thanks for the info!