https://dagster.io/ logo
Title
j

Joe

05/12/2023, 6:35 PM
If I have a new column in snowflake is there a way to delete the table and recreate it. I keep having an issue where my source data changes so I need to do that manually and wanted to do it in dagster. I don't mind dropping the table and recreating it everytime either. I am also not using partitions so trying to figure out why it is not dropping my tables
j

jamie

05/12/2023, 7:09 PM
hey @Joe we’re using a DELETE statement in the IO manager right now, so it’s deleting the rows but not dropping the entire table. We had a community contribution to change this, but it’s stagnated after a bit. I’ll revive it!
j

Joe

05/12/2023, 7:16 PM
thank you 🙂
I guess what I can do is create an Asset for each one of my assets that drops the table beforehand
f

Félix Tremblay

05/12/2023, 7:27 PM
@jamie can you please share the link to the contribution? I'm curious how it would handle partitioned tables
j

jamie

05/12/2023, 7:27 PM
yeah that should work as a work around, but definitely less than ideal. this issue kinda collides with other issues with doing all table mutations within a single transaction and how we handle adding columns for a partitioned asset. I need to think about all of these and come up with a unified strategy, so idk how quickly i’ll be able to turn a PR out
so the community contribution doesn’t handle adding columns for partitioned or non-partitioned. I think we could handle adding columns to non-partitioned by doing DROP instead of TRUNCATE, but adding columns to partitioned tables gets really complicated