https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
j

Jaap Langemeijer

07/05/2022, 9:11 AM
Hi #dagster-kubernetes! I am deploying dagster using the helm chart. I want to add some resources to my k8sRunLauncher, as otherwise my jobs get OOMKilled. I am using the following
values.yaml
for the runlauncher part:
Copy code
runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      resources:
        requests: 
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: 1
          memory: 4Gi
      envVars:
        - GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/key.json
      volumes:
        - name: gcloud-secrets
          secret:
            secretName: gcloud
      volumeMounts:
        - name: gcloud-secrets
          mountPath: /root/.config/gcloud
The
dagster-instance
configmap part looks good:
Copy code
run_launcher:
  module: dagster_k8s
  class: K8sRunLauncher
  config:
    load_incluster_config: true
    job_namespace: gww
    image_pull_policy: Always
    service_account_name: dagster
    dagster_home: "/opt/dagster/dagster_home"
    instance_config_map: "dagster-instance"
    postgres_password_secret: "dagster-postgresql-secret"
    env_vars:
      - GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/key.json
    volume_mounts:
      - mountPath: /root/.config/gcloud
        name: gcloud-secrets
    volumes:
      - name: gcloud-secrets
        secret:
          secretName: gcloud
    resources:
      limits:
        cpu: 1
        memory: 4Gi
      requests:
        cpu: 250m
        memory: 256Mi
Still, when describing a pod lauched from the dagit UI, I am not seeing my resource requirements back, while all the other settings (e.g. volumes/mounts, env vars etc.) are being set on the pod. I am out of ideas. Anyone has encountered this before? Thanks in advance!
after reading some issues above, I found out that the error comes from my user deployment overriding the
k8sRunLauncher
resources. Interesting, as I do not want my code repository to have the same resources as my runs.
j

johann

07/05/2022, 8:39 PM
Hi @Jaap Langemeijer, yeah this is tricky. Currently there’s two options: • disable
includeConfigInLaunchedRuns
, which configures the overriding. https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L268-L271 The downside is that things like secrets etc. will also not be copied over • override the override: you can specify resources per job using tags: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment
❤️ 1
I think we’ll consider not including resources in the
includeConfigInLaunchedRuns
override, it makes that you often wouldn’t want them to be the same
@Dagster Bot issue Support includeConfigInLaunchedRuns without overriding resources
d

Dagster Bot

07/05/2022, 8:42 PM
j

Jaap Langemeijer

07/06/2022, 7:24 AM
@johann thanks for the clear answer!
3 Views