is it possible to launch a run in a docker container and the

Question

is it possible to launch a run in a docker container and then have each fanned out op step be done in process or multi process I m not sure how the DockerRunLauncher interacts with the executor for each op step

daniel · Answer

Hi Danny doing it multi process is actually the default behavior if you use the DockerRunLauncher it still uses the default executor which is described here <https docs dagster io concepts ops jobs graphs job execution default job executor>

Danny Steffy · Answer

we have an `Engine Event` saying `Starting execution with step handler DockerStepHandler ` Does that launch a new container for each step How do I configure it not to do that for each step

daniel · Answer

I would only expect that to show up if you ve explicitly enabled the docker executor in your job

Danny Steffy · Answer

I have a defined job on a sensor that has that specified but does the UI use that job as well for the asset

daniel · Answer

I wouldn t expect it to if it s solely defined on the sensor any chance you can share the code

Danny Steffy · Answer

here s the graph of ops and asset definition from the graph ``` graph def full scored data set recruiter teams to score trained model chunk the recruiters to score and score them in batches result = chunking op recruiter teams to score map lambda batch generate score for ids batch trained model collect result = key to score recruiter teams to score map lambda key score data set data to score for key recruiter in batch=key trained model collect return append scores result graph asset = AssetsDefinition from graph full scored data set keys by input name= recruiter teams to score AssetKey recruiter teams to score trained model AssetKey trained model resource defs= mem mem io manager ```

Danny Steffy · Answer

here s the sensor definition ``` distributed scoring sensor define asset job distributed scoring job selection=AssetSelection groups distributed scoring executor def=docker executor ```

daniel · Answer

If you re then launching a job called distributed scoring job in dagit then i would expect that to use that executor config

Danny Steffy · Answer

I m launching it via the Materialize button in the UI

daniel · Answer

And what s It here precisely The distributed scoring job job

Danny Steffy · Answer

the `full scored data set` asset

daniel · Answer

Hmm I ll ask around about this with somebody who will know this for sure

Danny Steffy · Answer

thanks

Danny Steffy · Answer

our end goal is to have the run be spun up in a Docker container and have `data to score for keys` and `score data set` be run in process so we can use the memory IO manager we don t want to persist the data from `data to score for keys` as it s a lot of data to persist to disk If that s not possible I m guessing I would need a custom IO manager that could store the data in a file and then clean up the file after the downstream op ingests it

daniel · Answer

is there also a Definitions object

daniel · Answer

just double checking that you didn t define a default executor there

Danny Steffy · Answer

ah yep I have it in the Definitions object

daniel · Answer

ah that makes more sense

Danny Steffy · Answer

would removing that still have the run be launched in a Docker Container

Danny Steffy · Answer

and then each step would be the default multi process

daniel · Answer

yeah that s controlled by the DockerRunLauncher

Danny Steffy · Answer

excellent thanks for the help

Danny Steffy · Answer

ah so now the executor is only spinning up 4 processes at a time Is it possible to increase that default concurrency limitation for every run in the UI

daniel · Answer

Yeah take a look at the examples with max concurrent on that page <https docs dagster io concepts ops jobs graphs job execution default job executor>

Danny Steffy · Answer

right but can I pass that config to the Definitions object somehow Or would it be loaded in via the dagster yaml config

daniel · Answer

Yeah you can set default configuration by doing e g ```from dagster import multiprocess executor Definitions executor=multiprocess executor configured max concurrent 8 ```

Danny Steffy · Answer

ah prefect thanks so much

Danny Steffy · Answer

so I take it it s not possible to have the `data to score for key` op be a multiprocess executor but have the `score data set` op be an in process executor I was hoping to do that so that the `data to score for key` could be persisted only in memory until the `score data set` op is finished

Danny Steffy · Answer

if it s not possible would a custom IO manager that cleans up it s own files be the better solution

daniel · Answer

Splitting out executors per op isn t possible yet but it s something that s on our radar for a future improvement

daniel · Answer

What you re describing does sound more like an IO manager

Danny Steffy · Answer

would a method similar to this work then <https dagster slack com archives C01U954MEER p1677866136690939 thread ts=1677864919 402759 amp cid=C01U954MEER>

daniel · Answer

Hm this is a bit more out of my wheelhouse could you maybe make a new post for the IOManager piece and our OSS oncall who knows these things better can weigh in

Danny Steffy · Answer

absolutely thanks for your help with all of this Absolutely loving Dagster so far