If I have a new column in snowflake is there a way to delete the table and recreate it. I keep having an issue where my source data changes so I need to do that manually and wanted to do it in dagster. I don't mind dropping the table and recreating it everytime either. I am also not using partitions so trying to figure out why it is not dropping my tables
j
jamie
05/12/2023, 7:09 PM
hey @Joe we’re using a DELETE statement in the IO manager right now, so it’s deleting the rows but not dropping the entire table. We had a community contribution to change this, but it’s stagnated after a bit. I’ll revive it!
j
Joe
05/12/2023, 7:16 PM
thank you 🙂
Joe
05/12/2023, 7:16 PM
I guess what I can do is create an Asset for each one of my assets that drops the table beforehand
f
Félix Tremblay
05/12/2023, 7:27 PM
@jamie can you please share the link to the contribution? I'm curious how it would handle partitioned tables
j
jamie
05/12/2023, 7:27 PM
yeah that should work as a work around, but definitely less than ideal. this issue kinda collides with other issues with doing all table mutations within a single transaction and how we handle adding columns for a partitioned asset. I need to think about all of these and come up with a unified strategy, so idk how quickly i’ll be able to turn a PR out
jamie
05/12/2023, 7:28 PM
so the community contribution doesn’t handle adding columns for partitioned or non-partitioned. I think we could handle adding columns to non-partitioned by doing DROP instead of TRUNCATE, but adding columns to partitioned tables gets really complicated