https://dagster.io/ logo
#tools
Title
# tools
g

George Pearse

05/15/2022, 4:45 PM
Just started using DBT with dagster and love it. I have a v. conventional Data Warehouse in Postgres taking in a combination of Mongo and django API data. Does anyone have any recommendations for a tool to manage the warehouse schema? I'm familiar with flyway and alembic, but haven't used either extensively.
Can DBT do this?
s

sandy

05/16/2022, 2:53 PM
Hey George - I'm curious what you mean by "managing" the warehouse schema. Do you mean that you want to change the set of columns in a table when the columns that you've documented in your dbt schema.yml change?
g

George Pearse

05/16/2022, 3:42 PM
Yeah exactly, create migrations etc. Or is this not something you worry about so much in the Data Warehouse world? Think I keep getting confused with backend DB best practices.
Say I want to recreate my Data Warehouse in another storage solution, what's my starting point / how do I go about it?
s

sandy

05/16/2022, 10:57 PM
I think it is a real issue in the data warehouse world, but I'm not aware of software that's aimed at making it easy. One of the reasons is that you don't really need it in the simplest cases: it's typical to just drop tables and recreate them every time you want to refresh them. In the more complicated cases though, Alembic could be the way to go
1
g

George Pearse

05/23/2022, 1:14 PM
@sandy feel like this links in with the whole imperative vs declarative convo a bit
I guess nowadays for the DWH side of things you'd be normally mostly be using Airbyte + DBT in which case Airbyte makes the raw tables and DBT anything after that. So DDL management is somewhat less relevant.
s

sandy

06/08/2022, 4:58 PM
I think that's mostly true. An exception might be when it's too expensive to backfill an entire table, but tractable to backfill a single column
👍 1
38 Views