Hey everyone I m using both the DockerRunLauncher and the do

Question

Hey everyone I m using both the DockerRunLauncher and the docker executor Right now there is only one docker image that is used to build containers that execute job steps It would be great to have more than one image and to be able to specify which step uses which image Is it currently possible with Dagster Thanks pray

johann · Answer

Hi Meder I m curious what your use case is Are you looking to run a dagster op inside a different image Would you be running some other cli command

Meder Kamalov · Answer

Hi Johann Yes Some ops have dependency conflicts and running them in different images seemed like a solution Additionally and this doesn t have to be done via docker it would be awesome to be able to specify a worker for an op Some ops are memory intensive while others might require a GPU and so it would be great to be able to specify that somehow

johann · Answer

Got it Both of these features are things we have for K8s but haven t ported to Docker Can put them on the queue

johann · Answer

< Dagster Bot> issue Allow per step image and container config with docker executor

Dagster Bot · Answer

Created issue at <https github com dagster io dagster issues 8331>

Meder Kamalov · Answer

Thanks

Meder Kamalov · Answer

Hey < johann> I decided to give dagster k8s a go I can get the K8sRunLauncher working but how do I specify workers per op As mentioned above I would like to be able to select a specific image for it which in this case is a special worker But from what I see K8sRunLauncher uses the same type of image for all workers

johann · Answer

To use a different image for an op you ll want to use the `k8s job executor` Then you can specify an image in tags on each Op <https dagster slack com archives C01U954MEER p1644335565408589 thread ts=1644334246 680049 amp cid=C01U954MEER>