Currently running helm upgrade for dagster and we'...
# ask-community
a
Currently running helm upgrade for dagster and we're getting crashloopbackoff on the
dagster-postgresql
due to what appears to be a massive update in the postgres version?
Copy code
DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 14.6 (Debian 14.6-1.pgdg110+1).
2023-02-10 19:12:02.375 UTC [1] FATAL:  database files are incompatible with server
Is in the container logs before shutting down.
j
Hi Alec, yep we just bumped the default https://github.com/dagster-io/dagster/pull/12015. You can override this back to
9.6.21
Apologies for the breakage, we should at least highlight this more in the changelog
a
Is there a stable tag for the helm chart I should be using?
And any advice on migrating existing data?
Guess I will just pin Postgres for now
d
Hey Alec - this was an oversight, apologies for the breakage. We can show you how to set this in your values.yaml to be how it was before in the short-term
Adding this to your values.yaml should restore things to how it was before
Copy code
postgresql:
  image:
    repository: "library/postgres"
    tag: "9.6.21"
a
Thanks will do this now
d
I suspect the upgrade will involve running this command: https://www.postgresql.org/docs/9.6/pgupgrade.html (which is probably a good idea at some point as 9.6 is not currently officially supported by postgres, which is what prompted the upgrade - but you can do that at a time of your choosing)
a
since we are just on the cusp of our production instance needing to be stable, I may just purge it and start from scratch. Next time that won't do, though. It looks like you version the charts according to the dagster library itself, is that correct?
d
that's right - each weekly release has a new helm chart version. Despite your experience just now we don't intentionally push breaking changes as part of weekly releases other than for experimental features, this was an unfortunate oversight
a
@Alec Koumjian if you’re running in k8s in production, you may want to consider using an externally hosted database rather than running your database in the cluster
1
a
Probably on the roadmap. Although I've heard these days it's actually very possible to maintain a decent postgres config in k8s. Regardless, I will want to manage it more directly.
a
I would recommend running it externally for production workload. Just not worth the risk and the effort IMO