how to add max_concurrent_run for this job knowing that I'm using k8s_job_exectuor my_job = mygraph().to_job(executor = k8s_job_executor)

To limit the concurrency of runs for a job using the

k8s_job_executor

, you would not directly set a

max_concurrent

parameter on the executor itself, as this parameter is specific to the

multiprocess_executor

. Instead, you would manage run concurrency at the instance or run coordinator level. The documentation does not provide a direct example of setting a maximum number of concurrent runs for a job using the

k8s_job_executor

. However, you can control the concurrency of job runs by configuring the

QueuedRunCoordinator

in your Dagster instance settings. This coordinator allows you to limit the number of runs that are dequeued and launched concurrently. Here is an example of how you might configure the

QueuedRunCoordinator

in your

dagster.yaml

instance configuration file to limit the number of concurrent runs:

yaml run_coordinator:   module: dagster.core.run_coordinator   class: QueuedRunCoordinator   config:     max_concurrent_runs: 4  # Set the maximum number of concurrent runs

By setting

max_concurrent_runs

in the

QueuedRunCoordinator

configuration, you can limit the number of runs that are executed concurrently across the entire Dagster instance. This setting will apply to all jobs that are executed by the instance, not just a specific job. If you need to limit concurrency for a specific job, you would need to implement custom logic within your application to manage the queuing and launching of runs for that job, as the current documentation does not provide a built-in method to set concurrency limits on a per-job basis using the

k8s_job_executor

.

@Scout How do you setup this max concurrent runs using helm charts ?

To set up max concurrent runs using Helm charts for your Dagster deployment on Kubernetes, you would typically configure the

QueuedRunCoordinator

in the

dagster.yaml

file, which is part of the Dagster instance configuration. When deploying with Helm, you can provide custom values to the Helm chart, including the configuration for the Dagster instance. Here's a step-by-step guide on how to set the

max_concurrent_runs

using Helm: 1. Create a custom

values.yaml

file that includes the configuration for the

QueuedRunCoordinator

. You would specify the

max_concurrent_runs

under the

dagster-instance

key, which corresponds to the

dagster.yaml

settings. 2. In your custom

values.yaml

file, add the following configuration:

yaml dagster-instance:   configMaps:     - name: dagster-instance       data:         dagster.yaml: |           run_coordinator:             module: dagster.core.run_coordinator             class: QueuedRunCoordinator             config:               max_concurrent_runs: 4  # Set the maximum number of concurrent runs

3. Deploy or upgrade your Dagster release with Helm, providing your custom

values.yaml

file:

shell helm upgrade --install my-dagster-release dagster/dagster -f values.yaml

This command will apply your custom configuration, including the

max_concurrent_runs

setting, to your Dagster instance when the Helm chart is deployed. Please note that the exact structure of the

values.yaml

file and the Helm chart may vary depending on the version of the Dagster Helm chart you are using. Always refer to the specific documentation for your version of the Helm chart to ensure you are using the correct configuration keys and structure.