Craig Austin
10/13/2022, 10:03 PMop_tags
on @asset
to configure the k8s job is not working for me, but from looking at the code/docs, it seems like it should be combining anything in the pod_spec_config
key in "dagster-k8s/config"
w/ the dagster-generated job/pod config. I'm using the helm chart to install dagster (w/ run launcher K8sRunLauncher
) + the user-deployment helm chart to keep user code in a separate repo/namespace.
@asset(
op_tags={
"dagster-k8s/config": {
"pod_spec_config": {
"tolerations": [{
"key": "my-key",
"operator": "Exists",
"effect": "NoSchedule",
}],
},
},
},
)
def my_asset(_):
pass
Adam Bloom
10/14/2022, 12:11 AMdefine_asset_job
and including the tags thereCraig Austin
10/14/2022, 12:17 AM@repository
(at least as a default that could be overridden on assets) but that doesn't seem possible.chris
10/18/2022, 9:45 PMFrédéric Kaczynski
11/23/2022, 3:01 PM@asset(
op_tags={
"dagster-k8s/config": {
"container_config": {
"volume_mounts": [
{"name": "nfs-data", "mount_path": "/data"}
]
},
"pod_spec_config": {
"volumes": [
{"name": "nfs-data", "nfs": {...}}
]
}
}
}
)
def input_csv_data():
# ...
Having per-op k8s configuration is useful to give resources (RAM, GPU, ...) to a specific op when it's needed.