https://dagster.io/ logo
#ask-ai
Title
# ask-ai
m

Mikael Frosthage

03/12/2024, 9:14 AM
I need to limit the amount of concurrency. I am running open source using the helm chart. Do I just add the dagster.yaml file in the root of my project? Will dagster pick this up by convention.
s

Scout

03/12/2024, 9:14 AM
To limit concurrency in a Dagster deployment using the Helm chart, you would typically configure concurrency settings in the
values.yaml
file of your Helm chart rather than just placing a
dagster.yaml
file in the root of your project. The
values.yaml
file is used to set configuration options when you install or upgrade your Dagster deployment using Helm. This includes settings for the Dagster instance, such as run concurrency limits, which can be applied through the
QueuedRunCoordinator
. Here's an example of how you might configure concurrency limits in your
values.yaml
file:
Copy code
yaml runCoordinator:   enabled: true   type: QueuedRunCoordinator   config:     max_concurrent_runs: 10     tag_concurrency_limits:       - key: "my_concurrency_limit_tag"         limit: 5
In this example,
max_concurrent_runs
sets the maximum number of runs that are allowed to be in progress at once, and
tag_concurrency_limits
allows you to set limits based on specific tags. After updating your
values.yaml
file, you would apply these changes by upgrading your Helm release with the updated configuration. Please note that the exact structure of your
values.yaml
file and the available options may vary depending on the version of the Helm chart you are using. Always refer to the documentation specific to your version for the most accurate information. If you need to set concurrency limits at a more granular level, such as per code location, this is currently a feature request and not yet available in the open-source version of Dagster as of the last update from the documentation. You can follow the relevant GitHub issues and discussions for updates on this feature.
m

Mikael Frosthage

03/13/2024, 8:18 AM
@Scout I already have a limit of five on max_concurrent_runs.
s

Scout

03/13/2024, 8:18 AM
If you already have a
max_concurrent_runs
limit set to five in your
values.yaml
Helm chart configuration, this means that your Dagster instance will not execute more than five runs concurrently. This is a global setting that applies to all runs managed by the
QueuedRunCoordinator
. If you need to apply more specific concurrency limits, such as limiting concurrency based on certain tags or for specific assets, you can use the
tag_concurrency_limits
configuration. This allows you to set limits that are applied to runs with particular tags, providing more control over which runs can execute concurrently. For example, if you want to ensure that runs with a specific tag do not exceed a certain concurrency limit, you could add a
tag_concurrency_limits
entry like this:
Copy code
yaml runCoordinator:   enabled: true   type: QueuedRunCoordinator   config:     max_concurrent_runs: 5     tag_concurrency_limits:       - key: "resource"         limit: 2         value: "database"
In this configuration, the
max_concurrent_runs
is set to five, but there is an additional limit where only two runs tagged with
resource: database
can execute concurrently. Remember to apply your changes by upgrading your Helm release after modifying the
values.yaml
file. If you have further requirements for concurrency control that are not covered by these settings, please provide more details so that we can assist you better. It's also a good idea to keep an eye on the Dagster GitHub repository for any updates or new features related to concurrency management.
3 Views