https://dagster.io/ logo
#ask-ai
Title
# ask-ai
a

Andreas

03/13/2024, 11:42 AM
dagster max_concurrent in a job
s

Scout

03/13/2024, 11:42 AM
In Dagster, the
max_concurrent_runs
configuration setting is used to control the maximum number of runs that are allowed to be in progress at once. This setting is part of the
QueuedRunCoordinator
configuration, which is responsible for managing the run queue and launching runs. Here's how you can configure the
max_concurrent_runs
in your Dagster instance: 1. In your
dagster.yaml
file, under the
run_coordinator
section, you can specify the
max_concurrent_runs
setting. The default value is 10, but you can set it to -1 to disable the limit, to 0 to stop any runs from launching, or to any positive integer to set a specific limit. Here is an example configuration snippet from the documentation:
Copy code
yaml run_coordinator:   module: dagster._core.run_coordinator   class: QueuedRunCoordinator   config:     max_concurrent_runs: 5  # This sets the maximum concurrent runs to 5
2. If you want to apply more granular concurrency limits based on tags, you can use the
tag_concurrency_limits
setting within the same
run_coordinator
configuration. This allows you to set limits for runs with particular tags. Here is an example of how to use `tag_concurrency_limits`:
Copy code
yaml run_coordinator:   module: dagster._core.run_coordinator   class: QueuedRunCoordinator   config:     tag_concurrency_limits:       - key: "my_tag"         limit: 2  # This sets a limit of 2 concurrent runs for runs with the tag "my_tag"
Remember that these settings are part of the Dagster instance configuration and not specific to a single job. If you need to limit concurrency for a specific job, you would typically use tags and
tag_concurrency_limits
to achieve that. Please note that the
max_concurrent_runs
and
tag_concurrency_limits
settings are part of the
QueuedRunCoordinator
configuration, which requires the Dagster Daemon process to be running in order to manage and launch runs. If you have further questions or need assistance with a specific use case, feel free to ask!
2 Views