Hi all I am using Dagster backed by Postgres for instance st dagster #deployment-kubernetes

Hi all - I am using Dagster backed by Postgres for...

Arslan Aziz

03/28/2022, 9:53 PM

Hi all - I am using Dagster backed by Postgres for instance storage. Back when I first started with Dagster (0.12), I deployed on a VM and I had split my instance storage across three databases (runs, logs, and schedules), which I configured in my

dagster.yaml

. I am now moving my Dagster deployment to Kubernetes using the helm chart, however it seems that the helm chart only creates a single database to hold all instance data. Does anyone have tips on how my historical instance data (split across three databases) can be migrated with the helm deployment? Thanks!

johann

03/29/2022, 2:57 AM

Hi Arslan- a few recommendations: • The included in-cluster postgres deployment is good for trying things out, but we don’t necessarily recommend it for production. A lot of users point Dagster towards a hosted version for example • For the migration between databases, I think you should be able to replicate the tables from each of your 3 databases into the one new one

johann

03/29/2022, 2:59 AM

Something like https://stackoverflow.com/a/16708680/14656695 should do the trick- I would recommend using the same 0.12 version in your K8s cluster as you were running before to keep things simple, then potentially doing an upgrade after

❤️ 1

johann

03/29/2022, 3:07 AM

The table names for each db are listed as the first argument to db.Table in each of https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/storage/runs/schema.py https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/storage/event_log/schema.py https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/storage/schedules/schema.py

Arslan Aziz

03/29/2022, 2:13 PM

@johann I appreciate the information! I will give this a shot.

Open in Slack

Previous Next