https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
j

Javier Llorente Mañas

06/09/2022, 11:01 AM
Hey guys! My company is already using Dagster in Production, is working really well. So far in the initial design, we thought to have a separate K8s cluster for executing Dagster jobs triggered by different ML microservices deployed on a different cluster. We realize that Dagster consumed few k8s resources compared to our previous solution Kubeflow, so we are deciding to move everything to the same cluster. Do you have any recomendation for moving an existing Dagster environment within two different K8s clusters? Is there an issue if we have for a period of time two Dagit deployments pointing to the same database?
j

johann

06/09/2022, 4:49 PM
Is there an issue if we have for a period of time two Dagit deployments pointing to the same database
This isn’t necessarily a problem, we support Dagit replicas. With them running in different clusters though, you could definitely encounter edge cases (e.g. Dagit in cluster 1 will be able to show you runs in cluster 2, but wouldn’t be able to terminate them since that makes a k8s api call)
One thing that’s not supported is pointing two daemon processes at the same db, it will warnings and result in duplicate runs.
I’m wondering what the benefit of both Dagits pointed at one DB would be though? Unfortunately there’s not currently a great way to combine two Dagster databases, so unless you want to put in some work to migrate I think you’ll be losing run history from one cluster. If that’s the case, could you migrate just by adding the other set of job as a new deployment in the
dagster-user-deployments
list?
(Also I assumed that each cluster had a fully separate install of the Dagster helm chart, lmk if that’s not what you have going on)
j

Javier Llorente Mañas

06/10/2022, 8:20 AM
yeah so our problem it is we want to migrate Dagster from one cluster to another without losing jobs history
the helm chart it is the same but it is pointing to different configs
we are going to try to deploy everything at once
6 Views