I have a similar question. I would like my asset to append to an existing snowflake table (with a timestamp column) at the end of materialization instead of replacing an existing snowflake table. Is there a place in the examples or documentation that shows how to do this?
dagster bot responded by community 1
02/09/2023, 3:45 AM
I don't think dagster-snowflake io manager supports appending. However, you can do the write to snowflake in your asset op and then log the asset materialization directly. In other words, use any python tooling you like to write the data however you like into the database. Then, tell dagster what you did using the asset materialization.
02/09/2023, 3:40 PM
@Sean Davis How do you handle telling dagster what you did without it materializing the asset itself?
02/09/2023, 6:59 PM
Your op code materializes the asset (by appending to the database). The way you tell Dagster that you did that is by adding an AssetMaterialization to the op. That is the part that tells Dagster that the asset has actually been materialized.