The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hi Team! I have a question on concurrent runs. Does pipeline manage concurrent runs on it own? Can we control how many parallel threads for my pipeline?

Hi Manas, currently we support concurrency limits using Celery

Hi <@U015C9U9RLK>, can you please point me to relevant documentation?

If you create a worker pool with n worker processes, then at most n solids (with the given celery queue tag) will be able to run at once

We don’t currently support limits at the pipeline run level, but it’s something we’re discussing for 0.10.0. If it’s something you need, a github issue would be great

Here is a code example of specifying celery queue

<https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-celery/dagster_celery_tests/repo.py#L314-L318>

We don’t have much documentation around this unfortunately, it’s just Celery behavior that you can take advantage of through Dagster

It’s also worth mentioning if you aren’t using celery, the multi processes executor has concurrency limits as well
<https://docs.dagster.io/_apidocs/execution#dagster.multiprocess_executor>