https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alec Koumjian

02/02/2023, 3:46 PM
Hello! I have a job that needs a lot of ephemeral storage until the asset is loaded by the IOManager into cloud storage. I'm trying to figure out how to pass the custom kubernetes pod spec in to the
k8s_job_executor
Copy code
job_name = define_asset_job(
    "job_name",
    selection="populate_db",
    executor_def=k8s_job_executor.configured(
        {
            "volume_mounts": [{"mount_path": "/tmp/run/", "name": "run-volume"}],
            "volumes": [{"name": "run-volume", "empty_dir": {"size_limit": "1Ti"}}],
        }
    ),
)
This runs but I don't see the volume attributes in the job / pod.
d

daniel

02/02/2023, 3:52 PM
Hey Alec - that configuration would affect the spec for the pod launched for each individual step/op (the k8s_job_executor controls where each step executes). It sounds like you want to apply it to the pod for the whole run?
a

Alec Koumjian

02/02/2023, 3:53 PM
Technically either is fine. That job has a single asset in it. Knowing how to do either would be sufficient for my needs. But something is off about how I am sending the config because it is not picking it up.
wait a second, just grokked what you said
d

daniel

02/02/2023, 3:54 PM
Got it - in that case you may not need the k8s_job_executor at all (if there's only one op) What you can do is either stick that volume on every job on the instance: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#instance-level-kubernetes-configuration Or put it in a dagster-k8s/config tag on the individual job: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#per-job-or-per-op-kubernetes-configuration (so the argument to set in define_asset_job would be tags not executor_def)
a

Alec Koumjian

02/02/2023, 3:54 PM
Aha, so I've got a pod to manage the job but then also have a pod for the individual steps, in this case just one. That one has successfully mounted the volume
Yes, I don't need the k8s executor for this. Okay, thank you for helping me clear that up!
condagster 1
37 Views