i also don’t want to set

max_concurrent_runs

to 1, because I want to parallelize execution by default and only limit concurrency in this specific part of the pipeline

I think you can do this using the

tag_concurrency_limits

config value in the

run_coordinator

in

dagster.yaml

. https://docs.dagster.io/deployment/run-coordinator

Right now we offer run-level concurrency and op-level concurrency within a single run - we don't yet offer global op-level concurrency without setting up something like Celery, but we're actively discussing it here: https://github.com/dagster-io/dagster/issues/12470

@Danny Steffy I think this is actually exactly what I’m looking for! I’ve tested it with custom tags inserted at runtime and it’s behaving as I’d like. Now i’m having trouble ensuring that an asset job has specific tags…

Jack I just want to make sure you’re aware though that what Danny posted is still purely concurrency at the run level - you mentioned “only limit concurrency in this specific part of the pipeline” - we do not currently offer a feature that does that across multiple runs

thanks for that daniel - as long as I can tag specific jobs as “limited to 1 run at a time” and tag other jobs to be “go buck wild” I think that’s good enough for my purposes. Do you have any docs/tips on how to ensure that a specific asset job is given a run tag by default?

You should be able to supply a dictionary of tag key value pairs when creating the job

hm, ok, that’s what i thought, but it’s not working. let me debug

mapping is just a read-only dict. If you could pass along code that reproduces the problem we can take a look

ok lemme try to make a thing to reproduce, thanks for your help!