I have a simple pipeline but it took 1 minute betw...
# announcements
n
I have a simple pipeline but it took 1 minute between each solid
a
hmm this is very strange - looks like you were just using the default in process executor?
n
yes, and it's not really clear how to increase or change it
I try to use multiprocessing or dask but I think the configuration is not correct or not in the good place
a
Copy code
execution:
  multiprocess:
should be all you need to enable multiprocess - but there really shouldn’t be substantial delay between steps in the default in process executor
n
a
if you could - the runs table has a
...
button that includes a “Download Debug File” if you want to send it over
n
ok, I'll do that
I get it, it's better
as I load the definition from the file for the pipeline
Copy code
@pipeline(
    preset_defs=[
        PresetDefinition.from_files(
            "multiprocessing",
            environment_files=[file_relative_path(__file__, 'multiprocessing.yaml')],
        ),
    ]
)
I didn't see I could change it in the playground
however, there was an error in the documentation
1
it should be
Copy code
execution:
  multiprocess:
    config:
      max_concurrent: 4
Thank you very much!!!