Our Data Engineering group is sold on Dagster but its resources are limited. We're trying to get a couple other groups to buy in, but they might need to see it in use first. So here's my question: What does the maintenance and upkeep look like for Dagster once it's deployed on Kubernetes? Is it something that we need an app admin team to own? Or can an architect run a few lines via CLI now and then?
I know this is probably an "It depends" answer, so I'll try to answer questions, but any information you can provide would be much appreciated!
dagster bot responded by community 1
02/03/2023, 6:13 PM
Running on ECS so only partially applicable, but I’m a single person team. In my experience once it's up and running, the only dagster-specific maintenance I've had to do has been when there's an instance upgrade recommended on a new release (veeeery rare). Outside of that, you might add more gRPC locations but that's just changing a YAML and redeploying.
02/03/2023, 6:23 PM
My advice from someone who works with Dagster on a team of two (who manages the application, the infra, and the dag code), it’s possible using the Helm chart.
02/03/2023, 7:02 PM
I'm also a team of one. The helm chart works well and is reasonably well-documented. I haven't found Dagster to be particularly difficult to maintain or upgrade. Sane defaults on the helm chart allow jobs to run on an autoscaling cluster without really any additional configuration.
If you have a running kubenetes cluster, I'd suggest literally giving it a try. 'helm install' pretty much just works, though you'll need to add your own code later on. That process is a one-time thing and then just bump the docker container version and upgrade helm to get new code changes.
02/03/2023, 7:51 PM
This all sounds perfect. Thank you @Vinnie, @Dusty Shapiro, and @Sean Davis! 🎉