https://dagster.io/ logo
#ask-ai
Title
# ask-ai
t

Tomasz Tyżaj

02/20/2024, 10:47 AM
what are the default resources for dagster-job in kubernetes executor and where are they set and how can i change them
s

Scout

02/20/2024, 10:47 AM
In Dagster, when using the Kubernetes executor, the default resources for a job are not explicitly set by default. Instead, the resources used by a job will depend on the Kubernetes cluster's defaults or any limits set on the namespace or service account under which the job is running. However, you can specify custom resource requests and limits for your Dagster jobs when using the
k8s_job_executor
. This is done by using the
dagster-k8s/config
tag in your job definition to include Kubernetes-specific configuration, such as resource requests and limits for the containers that will run your job's steps. Here's an example from the documentation on how to set resource requests and limits for a job:
Copy code
python from dagster import job from dagster_k8s import k8s_job_executor  @job(executor_def=k8s_job_executor.configured({     "job_config": {         "container_config": {             "resources": {                 "requests": {"cpu": "500m", "memory": "64Mi"},                 "limits": {"cpu": "1000m", "memory": "256Mi"},             }         }     } })) def my_job():     ...
In this example, the
k8s_job_executor
is configured with a
job_config
that includes
container_config
specifying the resource requests and limits for the job's container. To change the default resources for a job, you would modify the
resources
dictionary within the
container_config
to include the desired CPU and memory requests and limits. These settings will then be applied to the Kubernetes pods that are created to run the steps of your job. Remember that these configurations are specific to the
k8s_job_executor
and will not apply if you are using a different executor. If you need to set default resources at a more global level, you would typically do this through Kubernetes configurations outside of Dagster, such as setting default resource requests and limits at the namespace level.
t

Tomasz Tyżaj

02/20/2024, 11:01 AM
is there some default for K8sRunLauncher ?
8 Views