The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hi everyone,
Is there there any built-in function which help avoid having multiple instances of the same job for the same partition running in parallel? Or, even better, avoid having multiple instances of the op that materializes the same asset partition?
Thanks.

<@U015C9U9RLK> - is this possible to accomplish with the QueuedRunCoordinator?

&gt; avoid having multiple instances of the same job for the same partition running in parallel
I haven’t thought about this use case, but yeah it should be possible using the run queue. We tag partition runs with `dagster/partition:&lt;name&gt;` , and the run queue has a nifty `applyLimitPerUniqueValue` option

Something like
```# dagster.yaml
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    tag_concurrency_limits:
      - key: "dagster/partition"
        value:
          applyLimitPerUniqueValue: true
        limit: 1```
should work. Note that this will restrict all runs on the instance with matching partition keys, including runs of different Jobs (if the partition sets had overlapping keys). It’s not currently possible to add an AND to the limit to filter it down to just the one parition set

&gt; even better, avoid having multiple instances of the op that materializes the same asset partition
op level concurrency limits are something we’ve prototyped a bunch but don’t currently support

Thanks <@U011CET83FG> <@U015C9U9RLK>
Do I need to send the tag `dagster/partition:&lt;name&gt;` when enqueueing  jobs? Or it's added by Dagster by default?

I have two partitioned assets, and it seemed that tag_concurrency_limits on `dagster/partition`  prevented me from materialising the same partition (e.g: 2022-08-30) for those two assets at the same time. I guess the uniqueness is checked on the partition_key only, not the asset_key.

That’s correct. We don’t currently offer matching on both tags at the same time. Feel free to submit an issue. It also may be possible to hack in a unique suffix into your partition keys, but that’s not great