Denis Maciel

06/01/2022, 1:31 PM
Hey folks, we are just getting started with Dagster and we are wondering if we should have two separate deployments of Dagster (dev and prod) or a single deployment where you would configure the jobs differently depending if they're running on dev or prod. Do you have any strong opinions about these approaches?
👍 1

Isaac Harris-Holt

06/01/2022, 1:45 PM
We use separate deployments to keep concerns completely separate (our dev/prod stacks run in different availability zones). We use an environment variable along with
when creating dagster resources to make sure we get the right credentials in each env

Denis Maciel

06/01/2022, 1:49 PM
Thanks, @Isaac Harris-Holt! That's helpful! I guess that's will also be our setup once we get a bit more experienced with Dagster. The benefit of a single deployment for us who is that it would allow to iterate/experiment more quickly. We have a monorepo and I just think that waiting for production deployments could slow us down. Also I don't want to bug devops too much right from the outset :)

Isaac Harris-Holt

06/01/2022, 1:50 PM
Well if you are using Docker or Kube, your actual code repos are separate, which helps with the quick iteration 🙂


06/01/2022, 2:10 PM
yeah you could consider having dev vs. prod code repos in one deployment - one danger of having everything in one code repo on one deployment is that if someone pushes a change that breaks a job compilation the whole deployment will come crashing down and no one can launch / view jobs until it's fixed. But that's also something that good automated testing should be able to catch.


06/01/2022, 4:30 PM
I deploy on k8s so yeah it's a dev deployment and a prod deployment

Frank Dekervel

06/02/2022, 7:16 AM
we use k8s, we deploy dagster and all our code using helm charts and skaffold on dev. with skaffold, build-push-deploy only takes seconds (provided that you have speedy docker registry).
and skaffold has a livecoding feature ("skaffold dev") that auto-syncs changes in your python files for running containers. This doesn't work if you use kubernetes run coordinator because it can only sync to running containers, it cannot live-update container images
this also means that we can deploy as many dev envs as we want