Hello, we have Dagster running on OSS (GKE Autopilot) and we are running into some trouble with docker images. We are using a custom tag and the

pullPolicy

is set to

Always

. When we do a docker push, dagster is not pulling the latest image. Has anyone faced this before? Any fix?

the custom tag is different every update or are you trying to do updates in place? If you are doing updates in to the same docker tag you need to do something to force all the pods to restart so that they will pull the new image

It's the same docker tag. We kind of guessed this. So, how does it work in the Hybrid Cloud setting? Does the agent we install make sure that pods are restarted? Any pointers what's the best way to restart all the pods?

Using a scheme where the tag changes when the code changes is the best way i would say. Hybrid Cloud would suffer the same problem if you are trying to update a tag in place.

I think on Cloud I was using

latest

tag and it would automatically take new image without me restarting pods..

ah ok, it depends on what how you are interacting with Cloud when you update the

latest

tag. Updating other pieces of metadata or explicitly asking for a redeploy would work.

i often recommend something like

$GIT_COMMIT_SHA-$TIMESTAMP

for an image, so that way 2 things happen. 1. you always know which commit is associated with your image 2. the change in the timestamp always re-triggers a pull

Thanks @alex and @Simon Frid! We did something like this..