dagster max_concurrent in a job

In Dagster, the

max_concurrent_runs

configuration setting is used to control the maximum number of runs that are allowed to be in progress at once. This setting is part of the

QueuedRunCoordinator

configuration, which is responsible for managing the run queue and launching runs. Here's how you can configure the

max_concurrent_runs

in your Dagster instance: 1. In your

dagster.yaml

file, under the

run_coordinator

section, you can specify the

max_concurrent_runs

setting. The default value is 10, but you can set it to -1 to disable the limit, to 0 to stop any runs from launching, or to any positive integer to set a specific limit. Here is an example configuration snippet from the documentation:

yaml run_coordinator:   module: dagster._core.run_coordinator   class: QueuedRunCoordinator   config:     max_concurrent_runs: 5  # This sets the maximum concurrent runs to 5

2. If you want to apply more granular concurrency limits based on tags, you can use the

tag_concurrency_limits

setting within the same

run_coordinator

configuration. This allows you to set limits for runs with particular tags. Here is an example of how to use `tag_concurrency_limits`:

yaml run_coordinator:   module: dagster._core.run_coordinator   class: QueuedRunCoordinator   config:     tag_concurrency_limits:       - key: "my_tag"         limit: 2  # This sets a limit of 2 concurrent runs for runs with the tag "my_tag"

Remember that these settings are part of the Dagster instance configuration and not specific to a single job. If you need to limit concurrency for a specific job, you would typically use tags and

tag_concurrency_limits

to achieve that. Please note that the

max_concurrent_runs

and

tag_concurrency_limits

settings are part of the

QueuedRunCoordinator

configuration, which requires the Dagster Daemon process to be running in order to manage and launch runs. If you have further questions or need assistance with a specific use case, feel free to ask!