Currently running helm upgrade for dagster and we re getting

Question

Currently running helm upgrade for dagster and we re getting crashloopbackoff on the `dagster postgresql` due to what appears to be a massive update in the postgres version ```DETAIL The data directory was initialized by PostgreSQL version 9 6 which is not compatible with this version 14 6 Debian 14 6 1 pgdg110+1 2023 02 10 19 12 02 375 UTC 1 FATAL database files are incompatible with server``` Is in the container logs before shutting down

johann · Answer

Hi Alec yep we just bumped the default <https github com dagster io dagster pull 12015> You can override this back to `9 6 21`

johann · Answer

Apologies for the breakage we should at least highlight this more in the changelog

Alec Koumjian · Answer

Is there a stable tag for the helm chart I should be using

Alec Koumjian · Answer

And any advice on migrating existing data

Alec Koumjian · Answer

Guess I will just pin Postgres for now

daniel · Answer

Hey Alec this was an oversight apologies for the breakage We can show you how to set this in your values yaml to be how it was before in the short term

daniel · Answer

Adding this to your values yaml should restore things to how it was before ```postgresql image repository library postgres tag 9 6 21 ```

Alec Koumjian · Answer

Thanks will do this now

daniel · Answer

I suspect the upgrade will involve running this command <https www postgresql org docs 9 6 pgupgrade html> which is probably a good idea at some point as 9 6 is not currently officially supported by postgres which is what prompted the upgrade but you can do that at a time of your choosing

Alec Koumjian · Answer

since we are just on the cusp of our production instance needing to be stable I may just purge it and start from scratch Next time that won t do though It looks like you version the charts according to the dagster library itself is that correct

daniel · Answer

that s right each weekly release has a new helm chart version Despite your experience just now we don t intentionally push breaking changes as part of weekly releases other than for experimental features this was an unfortunate oversight

Adam Bloom · Answer

< Alec Koumjian> if you re running in k8s in production you may want to consider using an externally hosted database rather than running your database in the cluster

Alec Koumjian · Answer

Probably on the roadmap Although I ve heard these days it s actually very possible to maintain a decent postgres config in k8s Regardless I will want to manage it more directly

Andrea Giardini · Answer

I would recommend running it externally for production workload Just not worth the risk and the effort IMO