19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#announcements
Title
n
Nicolas Ramy
10/20/2020, 3:27 PMI have a simple pipeline but it took 1 minute between each solid
a
alex
10/20/2020, 4:23 PMhmm this is very strange - looks like you were just using the default in process executor?
n
Nicolas Ramy
10/20/2020, 4:23 PMyes, and it's not really clear how to increase or change it
I try to use multiprocessing or dask but I think the configuration is not correct or not in the good place
a
alex
10/20/2020, 4:27 PM Copy code
execution:
multiprocess:should be all you need to enable multiprocess - but there really shouldn’t be substantial delay between steps in the default in process executor
n
Nicolas Ramy
10/20/2020, 4:28 PMthank you, I found it here https://docs.dagster.io/_modules/dagster/core/definitions/executor#
a
alex
10/20/2020, 4:29 PMif you could - the runs table has a
...n
Nicolas Ramy
10/20/2020, 4:29 PMok, I'll do that
I get it, it's better
as I load the definition from the file for the pipeline
Copy code
@pipeline(
preset_defs=[
PresetDefinition.from_files(
"multiprocessing",
environment_files=[file_relative_path(__file__, 'multiprocessing.yaml')],
),
]
)I didn't see I could change it in the playground
however, there was an error in the documentation
✅ 1
it should be
Copy code
execution:
multiprocess:
config:
max_concurrent: 4Thank you very much!!!