https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Averell

08/26/2022, 9:32 AM
Hi everyone, Is there there any built-in function which help avoid having multiple instances of the same job for the same partition running in parallel? Or, even better, avoid having multiple instances of the op that materializes the same asset partition? Thanks.
s

sandy

08/26/2022, 2:55 PM
@johann - is this possible to accomplish with the QueuedRunCoordinator?
j

johann

08/26/2022, 5:56 PM
avoid having multiple instances of the same job for the same partition running in parallel
I haven’t thought about this use case, but yeah it should be possible using the run queue. We tag partition runs with
dagster/partition:<name>
, and the run queue has a nifty
applyLimitPerUniqueValue
option
Something like
Copy code
# dagster.yaml
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    tag_concurrency_limits:
      - key: "dagster/partition"
        value:
          applyLimitPerUniqueValue: true
        limit: 1
should work. Note that this will restrict all runs on the instance with matching partition keys, including runs of different Jobs (if the partition sets had overlapping keys). It’s not currently possible to add an AND to the limit to filter it down to just the one parition set
even better, avoid having multiple instances of the op that materializes the same asset partition
op level concurrency limits are something we’ve prototyped a bunch but don’t currently support
a

Averell

08/29/2022, 9:48 AM
Thanks @sandy @johann Do I need to send the tag
dagster/partition:<name>
when enqueueing jobs? Or it's added by Dagster by default?
s

sandy

08/29/2022, 2:39 PM
It's added by default
a

Averell

08/29/2022, 10:19 PM
Thanks
I have two partitioned assets, and it seemed that tag_concurrency_limits on
dagster/partition
prevented me from materialising the same partition (e.g: 2022-08-30) for those two assets at the same time. I guess the uniqueness is checked on the partition_key only, not the asset_key.
j

johann

08/30/2022, 1:49 PM
That’s correct. We don’t currently offer matching on both tags at the same time. Feel free to submit an issue. It also may be possible to hack in a unique suffix into your partition keys, but that’s not great
❤️ 1
a

Averell

08/31/2022, 3:08 AM
Thanks @johann
18 Views