https://dagster.io/ logo
#dagster-support
Title
# dagster-support
j

John Lee

12/16/2022, 3:26 PM
Hi all, I have a question regarding user code deployments on k8s. Does anyone know why the image used for dagster runs would not update despite updates to the to the user code deployment subchart? I'm using the helm chart with dagster version 1.1.5 in all pods. Dagit is configured with:
Copy code
workspace:
    enabled: true
    servers:
      - host: "k8s-my-pipeline"
        port: 3030
        name: "my-pipeline"
When I update the image tag in the values.yaml of the subchart and redeploy it, the code server image updates correctly but the jobs launched from dagit still use the original image.
Copy code
deployments:
  - name: "k8s-my-pipeline"
    image:
      repository: "my_pipeline_image"
      tag: changed_tag
      pullPolicy: Always
    dagsterApiGrpcArgs:
      - "--python-file"
      - "/opt/my_pipeline/repository.py"
    port: 3030
    includeConfigInLaunchedRuns:
      enabled: True
a

Adam Bloom

12/16/2022, 3:30 PM
Were the jobs queued before or after updating the tag? The runner image tag is set when a job is queued, not started
j

John Lee

12/16/2022, 3:35 PM
The jobs were launched from the dagit dashboard after the update of the image tag in the subchart. I updated the user code deployment several times and launched jobs each time. It was only at the end that dagit showed that all runs had launched in the same container. I was making changes to the ops at the time and the expected results were observed, which confuses me a little. I am inferring that the same container was used based on the pods manifest and the dagit dashboard. Perhaps this inference is incorrect.
j

jamie

12/16/2022, 3:37 PM
cc @daniel
d

daniel

12/16/2022, 3:39 PM
Hi John - i'd expect changes to the image tag to be reflected as soon as you update, certainly within a minute or so
if there's a set of steps we can follow that reproduces what you're seeing we could double-check that that's the case
j

John Lee

12/16/2022, 3:44 PM
ok thanks. i'll work through the example deployment and try to make some minimal changes to observe the behavior.
aha... I found the source of the issue. I needed to refresh the user code deployment in dagit post redeploy. Then the new image used for subsequent runs. Thanks for the help.
d

daniel

12/16/2022, 4:25 PM
Got it - I would actually expect that to automatically update in dagit
As soon as the user code deployment restarts
but we can double check that that is still working as expected
j

John Lee

12/16/2022, 4:33 PM
🤷 It's happening reliably for me. I'm monitoring the new pod creation on k9s and following its successful deployment I am launching runs in dagit. I can launch several using the older image and as soon as i refresh dagit the new image is used. Other details that are likely extraneous are that memoization is enabled and there is a volume mounted and my browser is Firefox.
d

daniel

12/16/2022, 4:37 PM
When you say "refresh dagit" - you mean press Reload from the Workspace/Code Lcoations tab?
j

John Lee

12/16/2022, 4:40 PM
Apologies I was unclear. yes, I mean click the little arrow that when I hover over it says 'Reload location <user-code-deployment>' when I am viewing a job. A little pop then displays "Code location reloaded!"
d

daniel

12/16/2022, 4:44 PM
Are there possibly any error logs in your dagit pod related to subscriptions?
I'd be especially looking for an error like this:
Copy code
AttributeError: 'function' object has no attribute 'handle_event'
Ah you know what, I think a fix for this actually went out in 1.1.6
I expect it to start auto-updating if you upgrade
j

John Lee

12/16/2022, 6:24 PM
Great. That upgrade worked. It autoreloads now. And yes I did have that error on the dagit pod prior to the upgrade. Thanks for the extra info.
condagster 1
2 Views