https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Davis Kirkendall

07/25/2022, 8:50 PM
I'm running the dagster helm chart with resource limits and trying to override the limits for a single job. However, it doesn't seem to be working (the job container get's the default resource contraints set):
Copy code
@op(
    # This task is extremely inefficient and requires a lot of memory
    tags={
        "dagster-k8s/config": {
            "container_config": {
                "resources": {
                    "requests": {"cpu": "500m", "memory": "8Gi"},
                    "limits": {"cpu": "2", "memory": "12Gi"},
                }
            },
        }
    }
)
def run_op(context: OpExecutionContext) -> None:
    ...

my_asset = AssetsDefinition.from_op(
    run_op,
)
The run launcher config is set to:
Copy code
run_launcher:
  module: dagster_k8s
  class: K8sRunLauncher
  config:
    ...    
    resources:
      limits:
        cpu: '1'
        memory: 2Gi
      requests:
        cpu: 500m
        memory: 1Gi
    ...
Any ideas why this wouldn't work?
r

rex

07/25/2022, 9:15 PM
you should put this tag on the job, not the op
are you using the k8s executor?
d

Davis Kirkendall

07/25/2022, 9:18 PM
the job has multiple ops, and I don't really want to use these resources for every op of the job. kind of like it says in the documentation:
I was assuming that the run launcher as configured above in the helm chart would mean that I'm using the k8s executor right? Each op is definitely creating a new kubernetes job.
d

daniel

07/26/2022, 2:52 AM
its a little confusing, but there are two separate things at play here - the run launcher (where the run happens) and the executor (where each individual op/step happens). Using the K8sRunLauncher will ensure that the run happens in a k8s pod, but doesn't imply that every op will happen in its own pod. The default behavior if you don't specify an executor is for each op to execute in its own process, but still all in the same kubernetes pod
If each op (within a single dagster job) is creating a new kubernetes job, that would imply to that you're adding the k8s_job_executor somewhere. The tags that you have in the code that you posted will only kick in if you're using that executor.
if instead you want those resources to apply to the k8s pod where the run is happening, you would apply it to the job instead (not the ops)
👍 1
t

Timo

07/26/2022, 11:41 AM
Hi @Davis Kirkendall, again you ^^ I had AGAIN the exact same issue just some days ago, Have a look here: https://github.com/dagster-io/dagster/issues/8980. Also related here https://github.com/dagster-io/dagster/issues/8405
❤️ 1