https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
j

Javier Llorente Mañas

02/01/2022, 4:10 PM
Hey folks! I am looking into creating a k8s job configuration for a job using Volumes mounting a secret, how could I set these objects I am searching through the source code, but I do not have a clear picture of should I define volumes?
d

daniel

02/01/2022, 5:07 PM
Hi javier, you can set volumes using the volumes/volumeMounts keys in the helm chart here: https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L444-L463 - you may need to set them on both the run launcher and in the user-code-deployments section (the former is used when the run is launched, the latter is used when serving metadata about your jobs to dagit)
j

Javier Llorente Mañas

02/01/2022, 6:29 PM
Ok an there is any possibility to set up this volumes as part of the job generation in python code ?
d

daniel

02/01/2022, 6:31 PM
its possible to add tags to your jobs in python that configure the k8s job - you could use that to configure the container that is spun up when the job is launched and add the volumes there: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#job-or-op-kubernetes-configuration
j

Javier Llorente Mañas

02/01/2022, 6:32 PM
Ok I see thanks for you quick response
And do you know an example or at least which is the model use by the tags, it follows the standard k8s model?
Also I could use this tags in a job level aswell?
d

daniel

02/01/2022, 6:36 PM
yeah, they should be passed into a V1Container, V1PodSpec, etc. : https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-k8s/dagster_k8s/job.py#L556-L586 - and you should put them on the Dagster @job decorater, yeah (it won't actually work if you set it on the @op ) unless you are using a certain executor that runs each op in its own pod
j

Javier Llorente Mañas

02/01/2022, 7:16 PM
looking through the code I can imagine that something like this could work
Copy code
@job(tags={'dagster-k8s/config': {'container_config': {'resources': {'requests': { 'cpu': '1', 'memory': '500Mi' },}}, 
                                  'job_config':{'volume':[{'name': 'train_dir', 'mountPath': '/train_dir'}], 'volume_mounts':[{'name': 'train_dir', 'mountPath': '/train_dir'}]}}})
def train_model_job():
    pass
d

daniel

02/01/2022, 7:16 PM
I think it would be
volumes
and
volumeMounts
but otherwise yes i think that's right
or, sorry, you may be right
volume_mounts
my mistake
2 Views