Hi Team I have a question on concurrent runs Does pipeline m

Question

Hi Team I have a question on concurrent runs Does pipeline manage concurrent runs on it own Can we control how many parallel threads for my pipeline

johann · Answer

Hi Manas currently we support concurrency limits using Celery

Manas Jain · Answer

Hi < johann> can you please point me to relevant documentation

johann · Answer

If you create a worker pool with n worker processes then at most n solids with the given celery queue tag will be able to run at once

johann · Answer

We don t currently support limits at the pipeline run level but it s something we re discussing for 0 10 0 If it s something you need a github issue would be great

johann · Answer

Here is a code example of specifying celery queue

johann · Answer

johann · Answer

We don t have much documentation around this unfortunately it s just Celery behavior that you can take advantage of through Dagster

Manas Jain · Answer

< johann> thanks

johann · Answer

It s also worth mentioning if you aren t using celery the multi processes executor has concurrency limits as well <https docs dagster io apidocs execution dagster multiprocess executor>