Ken
05/30/2020, 12:08 AMLAUNCH_PIPELINE_EXECUTIOIN
to Dagit Graphql Server
• Dagit Graphql will create an ephemeral "dagster-run" job, that will run the pipeline as requested by the Graphql call.
• This "dagster-run" job creates a Pod, which by default will request for 100m of CPU resources.
Because the pipeline run can become very large and memory consuming, I want to set custom resources for the pod.
But, when I look at the code in /dagster/python_modules/libraries/dagster-k8s/dagster_k8s/job.py
where we create a job_container using kubernetes.client.V1Container()
,
There is no place to set "ResourceRequirement"
The V1Container() API does allow setting it, according to [1]
Is there a plan to support for this feature ?
[1] https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Container.md -> "resources"nate
05/30/2020, 12:22 AMnate
05/30/2020, 12:23 AMKen
05/30/2020, 12:25 AM