https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
e

Eldan Hamdani

02/28/2022, 12:18 PM
Hi, I created a tag for customizing the resources when a new pod spins up:
Copy code
tags={
        'dagster-k8s/config': {
            'container_config': {
                'resources': {
                    'requests': {'cpu': '4000m', 'memory': '19073Mi'},
                    'limits': {'cpu': '6000m', 'memory': '23841Mi'},
                }
            },
but I see that the pod which spins up failed after 4 min due to
OOMKilled
and if I describe the pod I see this resources config:
Copy code
Limits:
      cpu:     500m
      memory:  2560Mi
    Requests:
      cpu:     250m
      memory:  64Mi
do you know why it doesn’t work as expected?
????
d

daniel

02/28/2022, 2:21 PM
(Hi, re: your followup comment, to set expectations about response latency - most of the core team is usually working during business hours PST-EST. We'll try to get to most questions within 24 hours, often but not always sooner) Did you set the tags on the Dagster job or on a Dagster op? If you're using the K8sRunLauncher, the tags would need to be on the Dagster job
e

Eldan Hamdani

02/28/2022, 2:22 PM
it’s on ops
what do you mean on dagster job?
d

daniel

02/28/2022, 2:26 PM
the example here includes tags on a Dagster op and tags on a Dagster job: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#job-or-op-kubernetes-configuration Putting tags on an op will only change things if you're using something that runs each op in its own Kubernetes pod, like the k8s_job_executor. If you want the resource limits to apply to the whole run, it needs to go on the Dagster job like in the @job decorator in that example in the docs
1
e

Eldan Hamdani

02/28/2022, 2:27 PM
ohh ok
so the resources under job() is necessary
so why do I need the resources on the ops
?
d

daniel

02/28/2022, 2:28 PM
you don't need them on the ops unless you're using the k8s_job_executor. we'll update the docs there to be more clear
e

Eldan Hamdani

02/28/2022, 2:28 PM
so on my case I need just on the job?
d

daniel

02/28/2022, 2:29 PM
Exactly yeah
e

Eldan Hamdani

02/28/2022, 2:29 PM
I need to move the same tag to job?
d

daniel

02/28/2022, 2:29 PM
just like in the example I posted, yeah
Copy code
@job(
  tags = {
    'dagster-k8s/config': {
      'container_config': {
        'resources': {
          'requests': { 'cpu': '200m', 'memory': '32Mi' },
        }
      },
    }
  }
)
e

Eldan Hamdani

02/28/2022, 2:30 PM
but there is no limits
how does it make sense
d

daniel

02/28/2022, 2:31 PM
You can put limits there too - put exactly the tags you were using before, but on the job instead of the op
e

Eldan Hamdani

02/28/2022, 2:31 PM
ok got it
thank you!
condagster 1
ok it doesn’t work for job as well… @daniel
d

daniel

02/28/2022, 2:58 PM
Is the tag being applied to the run?
e

Eldan Hamdani

02/28/2022, 2:59 PM
what tag?
d

daniel

02/28/2022, 2:59 PM
The dagster-k8s/config tag that you applied to the dagster job
e

Eldan Hamdani

02/28/2022, 3:00 PM
it’s working cause I already have under the tag
nodeAffinity
which already works
but the resources section still not
d

daniel

02/28/2022, 3:05 PM
Can you check that the tag isn't somehow getting overridden in the launchpad? I've seen an issue once in a while that we are working on fixing where I needed to edit the tags for the run in the launchpad to get a new tag to stick
I had to go in and edit the tags (to remove the previous value) using this dialog to get my new tag change to actually be applied
o

Or Asher

02/28/2022, 3:14 PM
(from the same team as Eldan) is there a way to view the full tags assigned to a job? we clearly have some tags from the definition but cant see the complete values
d

daniel

02/28/2022, 3:17 PM
That dialog I posted just above has them (from the "Add tags"/"Edit tags" button in the launchpad) - there's a "Tags from definition" section and a custom tags section
and then previously launched runs should have all the tags listed in the runs list
o

Or Asher

02/28/2022, 3:27 PM
specifically k8s tags dont show up well and cant be expanded, so i cant be sure whats there
d

daniel

02/28/2022, 3:28 PM
yeah, that's good feedback that i'll pass along. It does look like your resource limits are being applied though, so I would expect them to be applied to the kubernetes pod for the Dagster job
e

Eldan Hamdani

02/28/2022, 5:00 PM
@daniel ok after many changes now I can change the resources, but the job is still failed with status
OOMKilled
after a few mins. the describe shows:
Copy code
Warning  FailedMount  44s (x2 over 44s)  kubelet            MountVolume.SetUp failed for volume "dagster-instance" : object "default"/"dagster-instance" not registered
  Warning  FailedMount  44s (x2 over 44s)  kubelet            MountVolume.SetUp failed for volume "kube-api-access-6jc7c" : object "default"/"kube-root-ca.crt" not registered
can you guide me what can be the issue here?
d

daniel

02/28/2022, 5:04 PM
I have a lot more expertise with the dagster-specific parts of the equation here (i.e. debugging why config isn't being set on the resulting k8s resources when it's set in dagster) than I do with how k8s responds to the pod configuration once it's set the way you expect
32 Views