Hi there What is a good strategy for no downtime upgrade dep

Question

Hi there What is a good strategy for no downtime upgrade deployments in Kubernetes with the OSS version of Dagster Our Kubernetes configuration currently has one instance of `dagster daemon` `dagit` and `dagster grpc` We restart all three of them at each upgrade of our application but this causes running pipelines to be left hanging the current op will finish but the pipeline will stay in RUNNING and never proceed to the next step it has to be terminated then restarted in dagit What would be a better way I was thinking of running `dagster daemon` and `dagit` from the official docker images so they are not tied to our release process and would stay up at all times except when we upgrade our Dagster version but `dagster grpc` would still need to restart since we run it from the Docker image that contains our source Would that help Kinda related is it supported to run a `daemon` and `dagit` on a given Dagster version but `grpc` on another

VxD · Answer

The documentation on <https docs dagster io deployment guides kubernetes deploying with helm> states gt Code location servers can be updated independently from other Dagster components including Dagit As a result updates to code locations can occur without causing downtime to any other code location or Dagit After updating if there is an error with any code location an error is surfaced for that code location within Dagit All other code locations and Dagit will still operate normally

VxD · Answer

I guess my issues come from the fact I am restarting both the daemon and the code location at around the same time If the code location is restarted even while pipelines are running will the daemon reconcile everything as long as it stays up the whole time

Joe · Answer

gt I was thinking of running `dagster daemon` and `dagit` from the official docker images so they are not tied to our release process and would stay up at all times except when we upgrade our Dagster version but `dagster grpc` would still need to restart since we run it from the Docker image that contains our source Would that help Yes that would avoid in flight runs being orphaned Currently it s best to avoid updating the daemon except when you want to upgrade your dagster version since that can t currently be done without downtime

VxD · Answer

Thanks Joe That s great info

Chris Histe · Answer

What happens to in progress runs when the dagster deamon is updated Are they killed

Joe · Answer

In progress runs will execute till completion