19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hey guys!
My company is already using Dagster in ...
# deployment-kubernetesj
Javier Llorente Mañas
06/09/2022, 11:01 AMHey guys!
My company is already using Dagster in Production, is working really well. So far in the initial design, we thought to have a separate K8s cluster for executing Dagster jobs triggered by different ML microservices deployed on a different cluster.
We realize that Dagster consumed few k8s resources compared to our previous solution Kubeflow, so we are deciding to move everything to the same cluster. Do you have any recomendation for moving an existing Dagster environment within two different K8s clusters? Is there an issue if we have for a period of time two Dagit deployments pointing to the same database?
j
johann
06/09/2022, 4:49 PMIs there an issue if we have for a period of time two Dagit deployments pointing to the same databaseThis isn’t necessarily a problem, we support Dagit replicas. With them running in different clusters though, you could definitely encounter edge cases (e.g. Dagit in cluster 1 will be able to show you runs in cluster 2, but wouldn’t be able to terminate them since that makes a k8s api call)
johann
06/09/2022, 4:50 PMOne thing that’s not supported is pointing two daemon processes at the same db, it will warnings and result in duplicate runs.
johann
06/09/2022, 4:53 PMI’m wondering what the benefit of both Dagits pointed at one DB would be though? Unfortunately there’s not currently a great way to combine two Dagster databases, so unless you want to put in some work to migrate I think you’ll be losing run history from one cluster.
If that’s the case, could you migrate just by adding the other set of job as a new deployment in the
dagster-user-deploymentsjohann
06/09/2022, 4:55 PM(Also I assumed that each cluster had a fully separate install of the Dagster helm chart, lmk if that’s not what you have going on)
j
Javier Llorente Mañas
06/10/2022, 8:20 AMyeah so our problem it is we want to migrate Dagster from one cluster to another without losing jobs history
Javier Llorente Mañas
06/10/2022, 8:20 AMthe helm chart it is the same but it is pointing to different configs
Javier Llorente Mañas
06/10/2022, 8:20 AMwe are going to try to deploy everything at once
6 Views