Currently running helm upgrade for dagster and we'...
# ask-community
Currently running helm upgrade for dagster and we're getting crashloopbackoff on the
due to what appears to be a massive update in the postgres version?
Copy code
DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 14.6 (Debian 14.6-1.pgdg110+1).
2023-02-10 19:12:02.375 UTC [1] FATAL:  database files are incompatible with server
Is in the container logs before shutting down.
Hi Alec, yep we just bumped the default You can override this back to
Apologies for the breakage, we should at least highlight this more in the changelog
Is there a stable tag for the helm chart I should be using?
And any advice on migrating existing data?
Guess I will just pin Postgres for now
Hey Alec - this was an oversight, apologies for the breakage. We can show you how to set this in your values.yaml to be how it was before in the short-term
Adding this to your values.yaml should restore things to how it was before
Copy code
    repository: "library/postgres"
    tag: "9.6.21"
Thanks will do this now
I suspect the upgrade will involve running this command: (which is probably a good idea at some point as 9.6 is not currently officially supported by postgres, which is what prompted the upgrade - but you can do that at a time of your choosing)
since we are just on the cusp of our production instance needing to be stable, I may just purge it and start from scratch. Next time that won't do, though. It looks like you version the charts according to the dagster library itself, is that correct?
that's right - each weekly release has a new helm chart version. Despite your experience just now we don't intentionally push breaking changes as part of weekly releases other than for experimental features, this was an unfortunate oversight
@Alec Koumjian if you’re running in k8s in production, you may want to consider using an externally hosted database rather than running your database in the cluster
Probably on the roadmap. Although I've heard these days it's actually very possible to maintain a decent postgres config in k8s. Regardless, I will want to manage it more directly.
I would recommend running it externally for production workload. Just not worth the risk and the effort IMO