https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
q

Quy

03/27/2022, 12:37 PM
Hi folks, i want to mount a volume to a dagster job, hence 1. using
dagster-k8s/config
and add
secret
to
/tmp
via
tags
2. add secret to heml values but when I check, there is no file I want. Does anyone know what I missed ? here is the way I configued
Copy code
download.to_job(
    name='download',
    tags={"dagster-k8s/config": {
        "container_config": {
            "volume_mounts": [
                {
                    "name": "volume1",
                    "mountPath": "/tmp/gcp.json". # I tried /tmp/ 
                },
            ]
        },
        "pod_spec_config": {
            "volumes": [
                {
                    "name": "volume1",
                    "secret": {
                        "secretName": "dagster-secret"
                    }
                }
            ]
        }
    }}
)
inside download function, I called
Copy code
mfiles = []
for root, dirs, files in os.walk("/tmp/"):
    for file in files:
        mfiles.append(file)
context.log.debug(f"onlyfiles: \n {mfiles}")
heml values I used
Copy code
runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      envSecrets:
        - name: dagster-secret
dagster-user-deployments:
  enabled: true
  enableSubchart: true
  deployments:
    - name: "user-code-example"
      image:
        repository: "my_registry"
        tag: stg
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/app/repository.py"
      port: 3030
      envSecrets:
        - name: dagster-secret
1
d

daniel

03/28/2022, 2:13 PM
Hi Quy, if you run 'kubectl describe' on your pod, does it have your volumes mounted in the way you expect?
q

Quy

03/28/2022, 3:24 PM
hi @daniel, may i know which pod i should describe?
I inspected a job, there is no mount volume that I am expecting
@daniel should I mount volume at
runLauncher
, and
dagster-user-deployments
in heml value?
d

daniel

03/28/2022, 4:34 PM
You can, but if you only need to access it within your ops, putting it the tags for the job should be fine too
If you go to the page for the run in dagit, there's a View Tags and Config button in the upper right - does it show the tags that you expected?
(if you hover over it it should show the tags)
q

Quy

03/28/2022, 8:54 PM
Hi @daniel , I actually put at tags in my code but can't mount volume
When I view tags in Dagit, doesn’t have anything related to
container_config
and
pod_spec_config
d

daniel

03/28/2022, 10:16 PM
OK, I think the mystery for us to solve is why those tags aren't getting applied
if you go to launch the job in the Launchpad in dagit, the tags that you set in code should be there
here is what i see in the launchpad for a run using the tags
if that's not there, you should try reloading your code in dagit from the Workspace tab
q

Quy

03/28/2022, 10:26 PM
I'm using version 0.14.6
Is there anyelse you want to extract log?
d

daniel

03/28/2022, 10:31 PM
did you see my tips above about the launchpad
q

Quy

03/28/2022, 10:56 PM
Yes i did. Besides, recreatimg dagit and user-repository pod as well
d

daniel

03/28/2022, 10:59 PM
So the tag doesn't appear in the launchpad, even though you are setting it in code
?
q

Quy

03/28/2022, 11:08 PM
yes correct
no related configs to tags is shown at dagit
d

daniel

03/28/2022, 11:16 PM
I meant the Launchpad tab in dagit where you enter your run config (before you launch the run) from the most recent screenshot I posted - which is different than the page in your screenshot, did you check the Launchpad too? If it's not there either, Is it still not there if you press the + button to open a new tab in the Launchpad?
q

Quy

03/28/2022, 11:25 PM
already created a new tab in the Launchpad
d

daniel

03/28/2022, 11:34 PM
Could you make some other change to that job I. your code to sanity check that any code changes are making it into dagit? For example, renaming the name argument to something other than download?
q

Quy

03/29/2022, 12:09 AM
i did it, doesn’t work. I was wondering if i should look logs at other places
d

daniel

03/29/2022, 12:16 AM
Ok, so to summarize our latest understanding of the problem - none of the changes you are making in your code are showing up in dagit?
What steps are you taking after making changes to your code to make them show up in dagit typically / how do you get your code deployed into production?
q

Quy

03/29/2022, 12:32 AM
none of the changes you are making in your code are showing up in dagit?
my code reflects in Dagit log as my expectation but not tags (as previous screenshot showed)
What steps are you taking after making changes to your code to make them show up in dagit typically
I changed function name,
download
to
download_graph
, add more log lines to an ops. note in this example, I’m using 1 silly op to generate log
how do you get your code deployed into production?
not sure what you meant, but I use helm 0.14.6 and adjust some values. I have test my modification with dagster’s
user-code-example
, it works well with
single_pod_job
, and
step_isolated_job
d

daniel

03/29/2022, 12:36 AM
Sorry, I may have misunderstood you earlier. When you changed the name of the job from download to download_graph, did that change show up in dagit? When you said “doesn't work” I assumed that meant that the change didn't show up.
q

Quy

03/29/2022, 12:45 AM
When you changed the name of the job from download to download_graph, did that change show up in dagit?
yes it showed up in dagit
d

daniel

03/29/2022, 12:45 AM
When I load this code in a local dagit:
Copy code
from dagster import graph, op


@op
def my_op():
    print("HI")


@graph
def download():
    my_op()


my_job = download.to_job(
    name="download",
    tags={
        "dagster-k8s/config": {
            "container_config": {
                "volume_mounts": [
                    {
                        "name": "volume1",
                        "mountPath": "/tmp/gcp.json",  # I tried /tmp/
                    }
                ]
            },
            "pod_spec_config": {
                "volumes": [{"name": "volume1", "secret": {"secretName": "dagster-secret"}}]
            },
        }
    },
)
This is what I see:
If you do the same thing do you get different results?
If I click on Edit tags... in the launchpad in the upper right, I see the tags listed under "Tags from definition"
q

Quy

03/29/2022, 12:49 AM
at my local machine, tags show up
d

daniel

03/29/2022, 12:51 AM
Are there any differences between your local machine and the dagit you have deployed? Different dagster versions maybe?
q

Quy

03/29/2022, 12:53 AM
i have checked, they are identical. i used kubectl forward port 9999 , while 3000 is my local machine
d

daniel

03/29/2022, 12:57 AM
It's really hard for me to come with an explanation why two dagits on the same dagster version would be treating the same code differently. Just to rule it out, you could try clearing out your browsers local storage to see if the Launchpad in prod is incorrectly caching something, but if that was it, creating a new tab in the Launchpad should have fixed it
q

Quy

03/29/2022, 1:06 AM
clear cached is the one approach I try most of the time 😅 i meant, I tried
Copy code
global:
  postgresqlSecretName: "dagster-stg-pg-password-secret"

fullnameOverride: "dagster-stg"
dagit:
  replicaCount: 1
  workspace:
    enabled: true
    servers:
      - host: "data-injector"
        port: 3030
      - host: "user-code-example"
        port: 3030

computeLogManager:
  type: S3ComputeLogManager
  config:
    s3ComputeLogManager:
      bucket: "dagster-stg"
      prefix: "dagster-stg-compute-io"
      useSsl: true
      skipEmptyFiles: true

dagster-user-deployments:
  enabled: true
  enableSubchart: true
  deployments:
    - name: "user-code-example"
      image:
        repository: "<http://docker.io/dagster/user-code-example|docker.io/dagster/user-code-example>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/example_project/example_repo/repo.py"
      port: 3030
      env: {}
      envSecrets: []

    - name: "data-injector"
      image:
        repository: "***"
        tag: stg
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/app/repository.py"
      port: 3030
      envSecrets:
        - name: dagster-stg-gcp-serviceaccount-secret

runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      envConfigMaps: []
      envSecrets:
        - name: dagster-stg-gcp-serviceaccount-secret
      envVars: []

postgresql:
  enabled: false
  postgresqlHost: "***"
  postgresqlUsername: dagster_stg_owner
  postgresqlDatabase: dagster_stg_db

generatePostgresqlPasswordSecret: false
generateCeleryConfigSecret: false

rabbitmq:
  enabled: false

redis:
  enabled: false

flower:
  enabled: false

ingress:
  enabled: false

dagsterDaemon:
  enabled: true

migrate:
  enabled: false

serviceAccount:
  create: true
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: "***"
here is the value
I have check python version at dagit is 3.7.8 while user responsitory 3.9.11. is that the case ?
d

daniel

03/29/2022, 2:47 PM
that sounds plausible - typically the user supplies the user repository, and that can be any supported python version
q

Quy

03/29/2022, 3:02 PM
is there any measure to recheck dagit or launcher caching code in k8s ?
d

daniel

03/29/2022, 3:10 PM
I'm not aware of any caching in dagit that would affect this (other than the browser caching we previously discussed), but you could triple-check by restarting dagit. There's definitely nothing that would persist across dagit restarts
You may need to reload the location from the workspace tab for code changes to appear, that should clear out anything that dagit knows about your code (restarting dagit would be a more blunt way to do the same thing, just to be sure)
q

Quy

04/04/2022, 3:33 PM
thanks, I resolved my problem. you are right, my code was cached
4 Views