https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alexander Whillas

02/01/2023, 9:34 PM
Are there Op level semaphores in Dagster?
I have an Op that needs a mutex across jobs (it hits a rate limited API that fails when overwhelmed). The rest of the dag is not restricted. Is it possible to do this with a resource or some other mechanisum?
d

daniel

02/01/2023, 9:40 PM
We have tag-based op concurrency limits that you can apply within a single run: https://docs.dagster.io/concepts/ops-jobs-graphs/job-execution#op-concurrency-limits
a

Alexander Whillas

02/01/2023, 9:48 PM
ah, but
These limits are only applied on a per-run basis
I need it across runs 😞
d

daniel

02/01/2023, 9:49 PM
for that your best bet is to either use celery: https://docs.dagster.io/deployment/guides#executing-dagster or use a combination of run level task concurrency and ops-within-a-run concurrency https://docs.dagster.io/deployment/run-coordinator#limiting-run-concurrency
a

Alexander Whillas

02/01/2023, 9:52 PM
yeah, we were limiting runs to 1 per pipeline but this mutex is needed across pipelines and runs and so the whole system came to a crawl as we could only run one thing at once becuase of one op used across pipelines.
i guess what i'm saying is we need more fine grained control other than whole runs.
So it says:
Note: These limits are only applied on a per-run basis. You can apply op concurrency limits across multiple runs using the
celery_executor
but how...?
d

daniel

02/01/2023, 10:02 PM
Oh sorry I meant to post this earlier: https://docs.dagster.io/deployment/guides/celery - but sent the parent page instead
👍 1
a

Alexander Whillas

02/01/2023, 10:20 PM
man, this is ugly and going to be a real pain to debug. Is this functionality going to be moved into Dagster at some point?
d

daniel

02/01/2023, 10:21 PM
we'd like to support global concurrency limits at some point, yeah - but I don't have a specific timeline for it unfortunately
🥺 1
29 Views