https://dagster.io/ logo
#ask-ai
Title
# ask-ai
b

Binoy Shah

03/12/2024, 4:32 AM
@Scout How do I provide custom kubernetes resource overrides for an asset, I need to increase the Memory and CPU Limits for it
s

Scout

03/12/2024, 4:32 AM
To provide custom Kubernetes resource overrides for an asset, you can set tags on the asset that specify the resource requirements. These tags are then used by the
k8s_job_executor
to apply the specified resources when the asset is materialized in its own Kubernetes pod. Here's an example from the documentation on how to set these tags:
Copy code
python from dagster import asset from dagster_k8s import k8s_job_executor  @asset(     op_tags={         "dagster-k8s/config": {             "container_config": {                 "resources": {                     "limits": {"cpu": "500m", "memory": "2560Mi"},                 },             },             "pod_spec_config": {                 "node_selector": {                     "disktype": "ssd"                 }             }         }     } ) def my_asset():     # Asset computation logic     ...
In this example, the
op_tags
parameter is used to set the Kubernetes configuration for the asset. The
container_config
section within the
dagster-k8s/config
tag specifies the resource limits for CPU and memory. You can adjust these values to meet the specific requirements of your asset. Remember that setting the
dagster-k8s/config
tag on an asset will only work if you're using the
k8s_job_executor
that materializes each asset in its own Kubernetes pod. If you're using the
K8sRunLauncher
that runs each job in its own pod, you would set the
dagster-k8s/config
tag at the job level instead. Please ensure that your Dagster deployment is configured to use the Kubernetes executor and that your Kubernetes cluster has the necessary resources available to satisfy the specified limits.
2 Views