I need to limit the amount of concurrency. I am running open source using the helm chart. Do I just add the dagster.yaml file in the root of my project? Will dagster pick this up by convention.

To limit concurrency in a Dagster deployment using the Helm chart, you would typically configure concurrency settings in the

values.yaml

file of your Helm chart rather than just placing a

dagster.yaml

file in the root of your project. The

values.yaml

file is used to set configuration options when you install or upgrade your Dagster deployment using Helm. This includes settings for the Dagster instance, such as run concurrency limits, which can be applied through the

QueuedRunCoordinator

. Here's an example of how you might configure concurrency limits in your

values.yaml

file:

yaml runCoordinator:   enabled: true   type: QueuedRunCoordinator   config:     max_concurrent_runs: 10     tag_concurrency_limits:       - key: "my_concurrency_limit_tag"         limit: 5

In this example,

max_concurrent_runs

sets the maximum number of runs that are allowed to be in progress at once, and

tag_concurrency_limits

allows you to set limits based on specific tags. After updating your

values.yaml

file, you would apply these changes by upgrading your Helm release with the updated configuration. Please note that the exact structure of your

values.yaml

file and the available options may vary depending on the version of the Helm chart you are using. Always refer to the documentation specific to your version for the most accurate information. If you need to set concurrency limits at a more granular level, such as per code location, this is currently a feature request and not yet available in the open-source version of Dagster as of the last update from the documentation. You can follow the relevant GitHub issues and discussions for updates on this feature.

@Scout I already have a limit of five on max_concurrent_runs.

If you already have a

max_concurrent_runs

limit set to five in your

values.yaml

Helm chart configuration, this means that your Dagster instance will not execute more than five runs concurrently. This is a global setting that applies to all runs managed by the

QueuedRunCoordinator

. If you need to apply more specific concurrency limits, such as limiting concurrency based on certain tags or for specific assets, you can use the

tag_concurrency_limits

configuration. This allows you to set limits that are applied to runs with particular tags, providing more control over which runs can execute concurrently. For example, if you want to ensure that runs with a specific tag do not exceed a certain concurrency limit, you could add a

tag_concurrency_limits

entry like this:

yaml runCoordinator:   enabled: true   type: QueuedRunCoordinator   config:     max_concurrent_runs: 5     tag_concurrency_limits:       - key: "resource"         limit: 2         value: "database"

In this configuration, the

max_concurrent_runs

is set to five, but there is an additional limit where only two runs tagged with

resource: database

can execute concurrently. Remember to apply your changes by upgrading your Helm release after modifying the

values.yaml

file. If you have further requirements for concurrency control that are not covered by these settings, please provide more details so that we can assist you better. It's also a good idea to keep an eye on the Dagster GitHub repository for any updates or new features related to concurrency management.