Hi folks, i want to mount a volume to a dagster job, hence 1. using

dagster-k8s/config

and add

secret

to

/tmp

via

tags

2. add secret to heml values but when I check, there is no file I want. Does anyone know what I missed ? here is the way I configued

download.to_job(
    name='download',
    tags={"dagster-k8s/config": {
        "container_config": {
            "volume_mounts": [
                {
                    "name": "volume1",
                    "mountPath": "/tmp/gcp.json". # I tried /tmp/ 
                },
            ]
        },
        "pod_spec_config": {
            "volumes": [
                {
                    "name": "volume1",
                    "secret": {
                        "secretName": "dagster-secret"
                    }
                }
            ]
        }
    }}
)

inside download function, I called

mfiles = []
for root, dirs, files in os.walk("/tmp/"):
    for file in files:
        mfiles.append(file)
context.log.debug(f"onlyfiles: \n {mfiles}")

heml values I used

runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      envSecrets:
        - name: dagster-secret
dagster-user-deployments:
  enabled: true
  enableSubchart: true
  deployments:
    - name: "user-code-example"
      image:
        repository: "my_registry"
        tag: stg
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/app/repository.py"
      port: 3030
      envSecrets:
        - name: dagster-secret

Hi Quy, if you run 'kubectl describe' on your pod, does it have your volumes mounted in the way you expect?

hi @daniel, may i know which pod i should describe?

You can, but if you only need to access it within your ops, putting it the tags for the job should be fine too

Hi @daniel , I actually put at tags in my code but can't mount volume

OK, I think the mystery for us to solve is why those tags aren't getting applied

I'm using version 0.14.6

did you see my tips above about the launchpad

Yes i did. Besides, recreatimg dagit and user-repository pod as well

So the tag doesn't appear in the launchpad, even though you are setting it in code

yes correct

I meant the Launchpad tab in dagit where you enter your run config (before you launch the run) from the most recent screenshot I posted - which is different than the page in your screenshot, did you check the Launchpad too? If it's not there either, Is it still not there if you press the + button to open a new tab in the Launchpad?

already created a new tab in the Launchpad

Could you make some other change to that job I. your code to sanity check that any code changes are making it into dagit? For example, renaming the name argument to something other than download?

i did it, doesn’t work. I was wondering if i should look logs at other places

Ok, so to summarize our latest understanding of the problem - none of the changes you are making in your code are showing up in dagit?

none of the changes you are making in your code are showing up in dagit?

my code reflects in Dagit log as my expectation but not tags (as previous screenshot showed)

What steps are you taking after making changes to your code to make them show up in dagit typically

I changed function name,

download

to

download_graph

, add more log lines to an ops. note in this example, I’m using 1 silly op to generate log

how do you get your code deployed into production?

not sure what you meant, but I use helm 0.14.6 and adjust some values. I have test my modification with dagster’s

user-code-example

, it works well with

single_pod_job

, and

step_isolated_job

Sorry, I may have misunderstood you earlier. When you changed the name of the job from download to download_graph, did that change show up in dagit? When you said “doesn't work” I assumed that meant that the change didn't show up.

When you changed the name of the job from download to download_graph, did that change show up in dagit?

yes it showed up in dagit

When I load this code in a local dagit:

from dagster import graph, op


@op
def my_op():
    print("HI")


@graph
def download():
    my_op()


my_job = download.to_job(
    name="download",
    tags={
        "dagster-k8s/config": {
            "container_config": {
                "volume_mounts": [
                    {
                        "name": "volume1",
                        "mountPath": "/tmp/gcp.json",  # I tried /tmp/
                    }
                ]
            },
            "pod_spec_config": {
                "volumes": [{"name": "volume1", "secret": {"secretName": "dagster-secret"}}]
            },
        }
    },
)

at my local machine, tags show up

Are there any differences between your local machine and the dagit you have deployed? Different dagster versions maybe?

i have checked, they are identical. i used kubectl forward port 9999 , while 3000 is my local machine

It's really hard for me to come with an explanation why two dagits on the same dagster version would be treating the same code differently. Just to rule it out, you could try clearing out your browsers local storage to see if the Launchpad in prod is incorrectly caching something, but if that was it, creating a new tab in the Launchpad should have fixed it

clear cached is the one approach I try most of the time 😅 i meant, I tried

that sounds plausible - typically the user supplies the user repository, and that can be any supported python version

is there any measure to recheck dagit or launcher caching code in k8s ?

I'm not aware of any caching in dagit that would affect this (other than the browser caching we previously discussed), but you could triple-check by restarting dagit. There's definitely nothing that would persist across dagit restarts

thanks, I resolved my problem. you are right, my code was cached