https://dagster.io/ logo
#announcements
Title
# announcements
m

Manas Jain

09/01/2020, 3:40 PM
Hi Team! I have a question on concurrent runs. Does pipeline manage concurrent runs on it own? Can we control how many parallel threads for my pipeline?
j

johann

09/01/2020, 3:43 PM
Hi Manas, currently we support concurrency limits using Celery
m

Manas Jain

09/01/2020, 3:43 PM
Hi @johann, can you please point me to relevant documentation?
j

johann

09/01/2020, 3:44 PM
If you create a worker pool with n worker processes, then at most n solids (with the given celery queue tag) will be able to run at once
We don’t currently support limits at the pipeline run level, but it’s something we’re discussing for 0.10.0. If it’s something you need, a github issue would be great
Here is a code example of specifying celery queue
We don’t have much documentation around this unfortunately, it’s just Celery behavior that you can take advantage of through Dagster
m

Manas Jain

09/01/2020, 3:56 PM
@johann thanks!
j

johann

09/01/2020, 4:22 PM
It’s also worth mentioning if you aren’t using celery, the multi processes executor has concurrency limits as well https://docs.dagster.io/_apidocs/execution#dagster.multiprocess_executor