Hi there What is a good strategy for no downtime upgrade dep dagster #deployment-kubernetes

Hi there! What is a good strategy for no-downtime ...

VxD

02/18/2023, 7:31 AM

Hi there! What is a good strategy for no-downtime upgrade deployments in Kubernetes with the OSS version of Dagster? Our Kubernetes configuration currently has one instance of

dagster-daemon

dagit

and

dagster-grpc

. We restart all three of them at each upgrade of our application, but this causes running pipelines to be left hanging (the current op will finish, but the pipeline will stay in RUNNING and never proceed to the next step;,it has to be terminated then restarted in dagit). What would be a better way? I was thinking of running

dagster-daemon

and

dagit

from the official docker images so they are not tied to our release process and would stay up at all times (except when we upgrade our Dagster version), but

dagster-grpc

would still need to restart since we run it from the Docker image that contains our source. Would that help? Kinda related: is it supported to run a

daemon

and

dagit

on a given Dagster version, but

grpc

on another?

VxD

02/18/2023, 9:21 AM

The documentation on https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm states:

Code location servers can be updated independently from other Dagster components, including Dagit. As a result, updates to code locations can occur without causing downtime to any other code location or Dagit. After updating, if there is an error with any code location, an error is surfaced for that code location within Dagit. All other code locations and Dagit will still operate normally.

VxD

02/18/2023, 9:22 AM

I guess my issues come from the fact I am restarting both the daemon and the code location at around the same time? If the code location is restarted even while pipelines are running, will the daemon reconcile everything as long as it stays up the whole time?

Joe

02/20/2023, 3:17 PM

I was thinking of running
dagster-daemon
and
dagit
from the official docker images so they are not tied to our release process and would stay up at all times (except when we upgrade our Dagster version), but
dagster-grpc
would still need to restart since we run it from the Docker image that contains our source. Would that help?

Yes, that would avoid in-flight runs being orphaned. Currently it's best to avoid updating the daemon (except when you want to upgrade your dagster version) since that can't currently be done without downtime.

VxD

02/20/2023, 4:04 PM

Thanks Joe! That’s great info.

Chris Histe

02/20/2023, 4:05 PM

What happens to in progress runs when the dagster deamon is updated? Are they killed?

Joe

02/20/2023, 4:15 PM

In progress runs will execute till completion.

👍 1

10 Views

Open in Slack

Previous Next