19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi everyone, firstly just want to say what an awes...
# announcementsh
Hugo Pedroso de Lima
08/27/2020, 11:18 AMHi everyone, firstly just want to say what an awesome product you have created here. I have a question regarding the celery-k8s-job-executor. Is it possible to set k8s pod limits and requests per pipeline config when using this executor? If so how would one do it? Would be great to be able to do this so we can specify for example gpu pod resources.
j
johann
08/27/2020, 2:14 PMHi Hugo, you can specify k8s resource requests and node affinities via solid tags
johann
08/27/2020, 2:15 PMExample from our release notes (we just added node affinities so the config isn’t in the docs yet):
New
dagster-k8s/configJobJobJobSpecPodSpecPodTemplateSpec Copy code
@solid(
tags = {
'dagster-k8s/config': {
'container_config': {
'resources': {
'requests': { 'cpu': '250m', 'memory': '64Mi' },
'limits': { 'cpu': '500m', 'memory': '2560Mi' },
}
},
'pod_template_spec_metadata': {
'annotations': { "<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>": "true"}
},
'pod_spec_config': {
'affinity': {
'nodeAffinity': {
'requiredDuringSchedulingIgnoredDuringExecution': {
'nodeSelectorTerms': [{
'matchExpressions': [{
'key': '<http://beta.kubernetes.io/os|beta.kubernetes.io/os>', 'operator': 'In', 'values': ['windows', 'linux'],
}]
}]
}
}
}
},
},
},
)
def my_solid(context):
<http://context.log.info|context.log.info>('running')johann
08/27/2020, 2:19 PMper pipeline configUnfortunately we currently just do this per solid. If you use the syntax on a pipeline tag, it will apply to the run coordinator pod. If you’d like to make a github issue for the feature that would be great!
h
Hugo Pedroso de Lima
08/27/2020, 5:28 PMthis is awesome, thanks for the reply, its even better at the solid level than in the pipeline config so this seems like it covers what I need
👍 1
3 Views