https://dagster.io/ logo
#ask-ai
Title
# ask-ai
b

BC A

03/08/2024, 2:20 PM
how to add max_concurrent_run for this job knowing that I'm using k8s_job_exectuor my_job = mygraph().to_job(executor = k8s_job_executor)
s

Scout

03/08/2024, 2:20 PM
To limit the concurrency of runs for a job using the
k8s_job_executor
, you would not directly set a
max_concurrent
parameter on the executor itself, as this parameter is specific to the
multiprocess_executor
. Instead, you would manage run concurrency at the instance or run coordinator level. The documentation does not provide a direct example of setting a maximum number of concurrent runs for a job using the
k8s_job_executor
. However, you can control the concurrency of job runs by configuring the
QueuedRunCoordinator
in your Dagster instance settings. This coordinator allows you to limit the number of runs that are dequeued and launched concurrently. Here is an example of how you might configure the
QueuedRunCoordinator
in your
dagster.yaml
instance configuration file to limit the number of concurrent runs:
Copy code
yaml run_coordinator:   module: dagster.core.run_coordinator   class: QueuedRunCoordinator   config:     max_concurrent_runs: 4  # Set the maximum number of concurrent runs
By setting
max_concurrent_runs
in the
QueuedRunCoordinator
configuration, you can limit the number of runs that are executed concurrently across the entire Dagster instance. This setting will apply to all jobs that are executed by the instance, not just a specific job. If you need to limit concurrency for a specific job, you would need to implement custom logic within your application to manage the queuing and launching of runs for that job, as the current documentation does not provide a built-in method to set concurrency limits on a per-job basis using the
k8s_job_executor
.
b

BC A

03/08/2024, 2:22 PM
@Scout How do you setup this max concurrent runs using helm charts ?
s

Scout

03/08/2024, 2:22 PM
To set up max concurrent runs using Helm charts for your Dagster deployment on Kubernetes, you would typically configure the
QueuedRunCoordinator
in the
dagster.yaml
file, which is part of the Dagster instance configuration. When deploying with Helm, you can provide custom values to the Helm chart, including the configuration for the Dagster instance. Here's a step-by-step guide on how to set the
max_concurrent_runs
using Helm: 1. Create a custom
values.yaml
file that includes the configuration for the
QueuedRunCoordinator
. You would specify the
max_concurrent_runs
under the
dagster-instance
key, which corresponds to the
dagster.yaml
settings. 2. In your custom
values.yaml
file, add the following configuration:
Copy code
yaml dagster-instance:   configMaps:     - name: dagster-instance       data:         dagster.yaml: |           run_coordinator:             module: dagster.core.run_coordinator             class: QueuedRunCoordinator             config:               max_concurrent_runs: 4  # Set the maximum number of concurrent runs
3. Deploy or upgrade your Dagster release with Helm, providing your custom
values.yaml
file:
Copy code
shell helm upgrade --install my-dagster-release dagster/dagster -f values.yaml
This command will apply your custom configuration, including the
max_concurrent_runs
setting, to your Dagster instance when the Helm chart is deployed. Please note that the exact structure of the
values.yaml
file and the Helm chart may vary depending on the version of the Dagster Helm chart you are using. Always refer to the specific documentation for your version of the Helm chart to ensure you are using the correct configuration keys and structure.
16 Views