Hi, Is dagster by default multithreaded/multiprocess vs dagit playground execution not? I'm seeing a big difference in execution time, but it could also be server at work vs vpn at home (data is smallish though 7 mb)
n
nate
05/04/2020, 2:39 PM
hi! I don’t recall off the top of my head what the default is in Dagit, but generally you can configure execution to be multiprocess on a per-pipeline-run basis by adding the following to your pipeline config, this may help:
Copy code
execution:
multiprocess:
config:
# use available num CPUs
max_concurrent: 0