Hi all, I have a question regarding user code deployments on k8s. Does anyone know why the image used for dagster runs would not update despite updates to the to the user code deployment subchart? I'm using the helm chart with dagster version 1.1.5 in all pods. Dagit is configured with:

workspace:
    enabled: true
    servers:
      - host: "k8s-my-pipeline"
        port: 3030
        name: "my-pipeline"

When I update the image tag in the values.yaml of the subchart and redeploy it, the code server image updates correctly but the jobs launched from dagit still use the original image.

deployments:
  - name: "k8s-my-pipeline"
    image:
      repository: "my_pipeline_image"
      tag: changed_tag
      pullPolicy: Always
    dagsterApiGrpcArgs:
      - "--python-file"
      - "/opt/my_pipeline/repository.py"
    port: 3030
    includeConfigInLaunchedRuns:
      enabled: True

Were the jobs queued before or after updating the tag? The runner image tag is set when a job is queued, not started

The jobs were launched from the dagit dashboard after the update of the image tag in the subchart. I updated the user code deployment several times and launched jobs each time. It was only at the end that dagit showed that all runs had launched in the same container. I was making changes to the ops at the time and the expected results were observed, which confuses me a little. I am inferring that the same container was used based on the pods manifest and the dagit dashboard. Perhaps this inference is incorrect.

cc @daniel

Hi John - i'd expect changes to the image tag to be reflected as soon as you update, certainly within a minute or so

ok thanks. i'll work through the example deployment and try to make some minimal changes to observe the behavior.

Got it - I would actually expect that to automatically update in dagit

🤷 It's happening reliably for me. I'm monitoring the new pod creation on k9s and following its successful deployment I am launching runs in dagit. I can launch several using the older image and as soon as i refresh dagit the new image is used. Other details that are likely extraneous are that memoization is enabled and there is a volume mounted and my browser is Firefox.

When you say "refresh dagit" - you mean press Reload from the Workspace/Code Lcoations tab?

Apologies I was unclear. yes, I mean click the little arrow that when I hover over it says 'Reload location <user-code-deployment>' when I am viewing a job. A little pop then displays "Code location reloaded!"

Are there possibly any error logs in your dagit pod related to subscriptions?

Great. That upgrade worked. It autoreloads now. And yes I did have that error on the dagit pod prior to the upgrade. Thanks for the extra info.