https://dagster.io/ logo
#ask-ai
Title
# ask-ai
a

Albert

02/29/2024, 4:31 PM
Can you give an example of a dagster.yaml with two or more tag_concurrency_limits entries?
s

Scout

02/29/2024, 4:31 PM
Certainly! Here's an example of a
dagster.yaml
configuration file with two
tag_concurrency_limits
entries:
Copy code
yaml
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    tag_concurrency_limits:
      - key: "database"
        value: "redshift"
        limit: 2
      - key: "use-case"
        value:
          applyLimitPerUniqueValue: true
        limit: 3
In this example, the first entry specifies that there can be at most two concurrent runs with the tag
database
set to
redshift
. The second entry uses
applyLimitPerUniqueValue: true
to apply separate limits to each unique value of the
use-case
tag, with a limit of three concurrent runs for each unique value. Limiting concurrency in data pipelines |... button It's unclear how/where I should configur... button AutoMaterializePolicy not working for pa... button Controlling asset backfill concurrency button Thank you for your feedback!
a

Albert

02/29/2024, 4:33 PM
waht does the value part of the entry mean? I didn’t realize it could be something other than a string
@Scout what other configurations for value can be applied other than a string or this
applylimitperuniquevalue
s

Scout

02/29/2024, 4:35 PM
The documentation provides an example of using
applyLimitPerUniqueValue
to apply separate limits for each unique value of a tag. However, it does not explicitly mention other configurations for the
value
field within
tag_concurrency_limits
beyond a string or the
applyLimitPerUniqueValue
setting. The
value
field is typically used to match a specific tag value when you want to apply a concurrency limit to runs with that specific tag. When
applyLimitPerUniqueValue
is set to
true
, it indicates that the concurrency limit should be applied separately to each unique value of the tag specified by the
key
. If you need to apply different types of configurations for the
value
field, it would be best to refer to the latest Dagster documentation or release notes for any updates or additional features that may have been introduced after my knowledge cutoff date. As of my last update, the
value
field is used as described above. Tag values should be allowed if they are... button add to_config_str method to asset key button Allow any valid Pydantic field type to b... button Dagster Docs button Thank you for your feedback!
a

Albert

02/29/2024, 4:38 PM
@Scout I have multiple Workspaces configured and I would like the tag_concurrency_limits to be respected across all workspaces. It seems right now the concurrency is only for one workspace, how can I get it to work across all my workspaces?
s

Scout

02/29/2024, 4:38 PM
As of my last update,
tag_concurrency_limits
are applied within the scope of a Dagster instance. If you have multiple workspaces configured, each workspace may correspond to a separate Dagster instance, and
tag_concurrency_limits
would be applied independently within each instance. To have
tag_concurrency_limits
respected across all workspaces, you would need to ensure that all workspaces are connected to the same Dagster instance. This means that they would share the same
dagster.yaml
configuration file, including the
run_coordinator
configuration that specifies the
tag_concurrency_limits
. If your workspaces are indeed connected to different Dagster instances and you want to enforce global concurrency limits across these instances, this would require a feature that supports distributed concurrency control, which is not covered in the documentation as of my last update. If this is a critical requirement for your use case, you might consider the following approaches: 1. Centralized Dagster Instance: Reconfigure your workspaces to connect to a single Dagster instance that manages runs for all workspaces. This would naturally enforce the concurrency limits across all workspaces. 2. Custom Solution: Implement a custom solution outside of Dagster that manages the dispatching of runs to different instances based on global concurrency rules. This would likely involve a separate service or coordination layer that tracks active runs across instances and enforces limits before dispatching new runs. 3. Feature Request: If Dagster does not currently support this feature, you could file a feature request on the Dagster GitHub repository. The Dagster team and community could then consider this for future releases. For the most current information and potential new features that may address this need, please check the latest Dagster documentation or reach out to the Dagster community for assistance. Controlling asset backfill concurrency button Concurrency confusion with jobs button Limiting concurrency in data pipelines |... button Concurrency Controls at Code Location Sc... button Thank you for your feedback!