I'd like to say that I really like the new assets-related features! That's amazing! Thanks you for your great job!
I'm in process of migration from ops to assets and some interesting question appeared: were there any attempts to represent DB schema migrations as versioned assets? I'm thinking about Alembic, but it seems too complex from the first look.
01/10/2023, 4:47 PM
Hi Mykola - this is something that I've thought about a little bit, but I haven't seen people use Dagster for it. Do you have an idea of what functionality you'd be looking for in particular?
01/11/2023, 12:28 PM
The usecase is very simple and common:
• destination asset is a partitioned table in relational db
• at a very beginning the table may not be created, so I'd like depend on an asset which creates the table for me (as its materialization)
• at some point in time you may need to add some field/indexes to the table, so I'd like to migration happen due to asset "code version" change, where the version may be something like Alembic migration version hash (but as I said I'm not sure about the Alembic itself, i'm not an expert in it)
This is not my idea btw. I've met it somewhere in Dagster documentation.
01/11/2023, 4:18 PM
Got it - that makes sense. Would you want to backfill the values in that column somehow? Or just have the column there for when you're inserting new data?