Hey folks, we are just getting started with Dagste...
# random
Hey folks, we are just getting started with Dagster and we are wondering if we should have two separate deployments of Dagster (dev and prod) or a single deployment where you would configure the jobs differently depending if they're running on dev or prod. Do you have any strong opinions about these approaches?
👍 1
We use separate deployments to keep concerns completely separate (our dev/prod stacks run in different availability zones). We use an environment variable along with
when creating dagster resources to make sure we get the right credentials in each env
Thanks, @Isaac Harris-Holt! That's helpful! I guess that's will also be our setup once we get a bit more experienced with Dagster. The benefit of a single deployment for us who is that it would allow to iterate/experiment more quickly. We have a monorepo and I just think that waiting for production deployments could slow us down. Also I don't want to bug devops too much right from the outset :)
Well if you are using Docker or Kube, your actual code repos are separate, which helps with the quick iteration 🙂
yeah you could consider having dev vs. prod code repos in one deployment - one danger of having everything in one code repo on one deployment is that if someone pushes a change that breaks a job compilation the whole deployment will come crashing down and no one can launch / view jobs until it's fixed. But that's also something that good automated testing should be able to catch.
I deploy on k8s so yeah it's a dev deployment and a prod deployment
we use k8s, we deploy dagster and all our code using helm charts and skaffold on dev. with skaffold, build-push-deploy only takes seconds (provided that you have speedy docker registry).
and skaffold has a livecoding feature ("skaffold dev") that auto-syncs changes in your python files for running containers. This doesn't work if you use kubernetes run coordinator because it can only sync to running containers, it cannot live-update container images
this also means that we can deploy as many dev envs as we want