Hello, I would like to setup concurrency limit on a resource-level. My current understanding is that concurrency limit is setup based on job-level tags. Is there a way to add run tags based on required resources (specific API) or assets (that are generated by these API) to limit concurrent resource usage/calls ? Thank you

Hi lucas - as of yesterday we also support setting concurrency limits on specific assets or ops, but only within a single run: https://docs.dagster.io/concepts/ops-jobs-graphs/job-execution#op-concurrency-limits Would that work for your use case?

Hello Daniel, Thank you for the quick answer. It is indeed great for single run. However, if I wanted to enforce concurrency limit across all runs, how should I proceed ?

right now for that you would need to use celery https://docs.dagster.io/deployment/guides/celery

It is indeed pretty close. One of the issue I see is that I am not able to enforce tags on ad-hoc materialization runs which mean run-level concurrency is not applied for these runs. Is there any way to disable ad-hoc materialization (all asset materialization must be triggered from a job) ?

Are you applying those tags to the job in code? That doesn’t strictly enforce it, but somebody would have to work pretty hard to override it when materializing the run (by overriding the tag value)

Yes. I've created jobs per asset group with tag defining which resource is being consumed/called by this group. When materializing assets from jobs, run-level concurrency limit is enforced to prevent high concurrency on my resource. If assets are generated outside of my jobs (ad hoc materialization), runs are triggered without tags and thus are not limited.