I would like to setup concurrency limit on a resource-level.
My current understanding is that concurrency limit is setup based on job-level tags.
Is there a way to add run tags based on required resources (specific API) or assets (that are generated by these API) to limit concurrent resource usage/calls ?
Which has its own tradeoffs - we'd like to support a native global op limit in the future, but we don't currently
with a combination of run-level concurrency limits and op-level concurrency limits within a single run, you can get pretty close though
01/06/2023, 5:56 PM
It is indeed pretty close.
One of the issue I see is that I am not able to enforce tags on ad-hoc materialization runs which mean run-level concurrency is not applied for these runs.
Is there any way to disable ad-hoc materialization (all asset materialization must be triggered from a job) ?
01/06/2023, 6:36 PM
Are you applying those tags to the job in code? That doesn’t strictly enforce it, but somebody would have to work pretty hard to override it when materializing the run (by overriding the tag value)
01/06/2023, 7:26 PM
Yes. I've created jobs per asset group with tag defining which resource is being consumed/called by this group. When materializing assets from jobs, run-level concurrency limit is enforced to prevent high concurrency on my resource.
If assets are generated outside of my jobs (ad hoc materialization), runs are triggered without tags and thus are not limited.