The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Are there Op level semaphores in Dagster?

I have an Op that needs a mutex across jobs (it hits a rate limited API that fails when overwhelmed). The rest of the dag is not restricted. Is it possible to do this with a resource or some other mechanisum?

We have tag-based op concurrency limits that you can apply within a single run: <https://docs.dagster.io/concepts/ops-jobs-graphs/job-execution#op-concurrency-limits>

ah, but
&gt; These limits are only applied on a per-run basis
I need it across runs :disappointed:

for that your best bet is to either use celery: <https://docs.dagster.io/deployment/guides#executing-dagster>
or use a combination of run level task concurrency and ops-within-a-run concurrency <https://docs.dagster.io/deployment/run-coordinator#limiting-run-concurrency>

yeah, we were limiting runs to 1 per pipeline but this mutex is needed across pipelines and runs and so the whole system came to a crawl as we could only run one thing at once becuase of one op used across pipelines.

i guess what i'm saying is we need more fine grained control other than whole runs.

So it says:
&gt; Note: These limits are only applied on a per-run basis. You can apply op concurrency limits across multiple runs using the `celery_executor`
but how...?

Oh sorry I meant to post this earlier: <https://docs.dagster.io/deployment/guides/celery> - but sent the parent page instead

man, this is ugly and going to be a real pain to debug. Is this functionality going to be moved into Dagster at some point?

we'd like to support global concurrency limits at some point, yeah - but I don't have a specific timeline for it unfortunately