Can you give an example of a dagster.yaml with two or more tag_concurrency_limits entries?

Certainly! Here's an example of a

dagster.yaml

configuration file with two

tag_concurrency_limits

entries:

yaml
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    tag_concurrency_limits:
      - key: "database"
        value: "redshift"
        limit: 2
      - key: "use-case"
        value:
          applyLimitPerUniqueValue: true
        limit: 3

In this example, the first entry specifies that there can be at most two concurrent runs with the tag

database

set to

redshift

. The second entry uses

applyLimitPerUniqueValue: true

to apply separate limits to each unique value of the

use-case

tag, with a limit of three concurrent runs for each unique value. Limiting concurrency in data pipelines |... button It's unclear how/where I should configur... button AutoMaterializePolicy not working for pa... button Controlling asset backfill concurrency button Thank you for your feedback!

waht does the value part of the entry mean? I didn’t realize it could be something other than a string

The documentation provides an example of using

applyLimitPerUniqueValue

to apply separate limits for each unique value of a tag. However, it does not explicitly mention other configurations for the

value

field within

tag_concurrency_limits

beyond a string or the

applyLimitPerUniqueValue

setting. The

value

field is typically used to match a specific tag value when you want to apply a concurrency limit to runs with that specific tag. When

applyLimitPerUniqueValue

is set to

true

, it indicates that the concurrency limit should be applied separately to each unique value of the tag specified by the

key

. If you need to apply different types of configurations for the

value

field, it would be best to refer to the latest Dagster documentation or release notes for any updates or additional features that may have been introduced after my knowledge cutoff date. As of my last update, the

value

field is used as described above. Tag values should be allowed if they are... button add to_config_str method to asset key button Allow any valid Pydantic field type to b... button Dagster Docs button Thank you for your feedback!

@Scout I have multiple Workspaces configured and I would like the tag_concurrency_limits to be respected across all workspaces. It seems right now the concurrency is only for one workspace, how can I get it to work across all my workspaces?

As of my last update,

tag_concurrency_limits

are applied within the scope of a Dagster instance. If you have multiple workspaces configured, each workspace may correspond to a separate Dagster instance, and

tag_concurrency_limits

would be applied independently within each instance. To have

tag_concurrency_limits

respected across all workspaces, you would need to ensure that all workspaces are connected to the same Dagster instance. This means that they would share the same

dagster.yaml

configuration file, including the

run_coordinator

configuration that specifies the

tag_concurrency_limits

. If your workspaces are indeed connected to different Dagster instances and you want to enforce global concurrency limits across these instances, this would require a feature that supports distributed concurrency control, which is not covered in the documentation as of my last update. If this is a critical requirement for your use case, you might consider the following approaches: 1. Centralized Dagster Instance: Reconfigure your workspaces to connect to a single Dagster instance that manages runs for all workspaces. This would naturally enforce the concurrency limits across all workspaces. 2. Custom Solution: Implement a custom solution outside of Dagster that manages the dispatching of runs to different instances based on global concurrency rules. This would likely involve a separate service or coordination layer that tracks active runs across instances and enforces limits before dispatching new runs. 3. Feature Request: If Dagster does not currently support this feature, you could file a feature request on the Dagster GitHub repository. The Dagster team and community could then consider this for future releases. For the most current information and potential new features that may address this need, please check the latest Dagster documentation or reach out to the Dagster community for assistance. Controlling asset backfill concurrency button Concurrency confusion with jobs button Limiting concurrency in data pipelines |... button Concurrency Controls at Code Location Sc... button Thank you for your feedback!