Bojan11/04/2022, 2:11 PM
@asset( key_prefix=["my_schema"] # will be used as the schema in snowflake ) def my_table() -> pd.DataFrame: # the name of the asset will be the table name ...
James Hale11/04/2022, 2:17 PM
Bojan11/04/2022, 2:18 PM
James Hale11/04/2022, 2:19 PM
Bojan11/04/2022, 2:26 PM
@asset( key_prefix=["my_schema"] # will be used as the schema in snowflake ) def scrub_my_table() -> pd.DataFrame: return my_table.dropna() # This obv doesn't work but how would i read in the my_table initially ...
James Hale11/04/2022, 2:38 PM
Bojan11/04/2022, 6:36 PM
but i guess i haven’t figured out how to create and return the table from an asset using snowflake io manager
it would be quite helpful !
Create an asset to represent the table and pass it downstream.
James Hale11/06/2022, 7:50 PM
as: 1. Store the
output yielded by the software-defined asset as a JSON temp file 2. Create a table on Snowflake if it doesn't exist 3. Create a stage on Snowflake if it doesn't exist 4.
the file to the stage 5.
the contents of the file into the table You can see that IO manager here - it's implemented with SQLAlchemy: https://gist.github.com/jayhale/c5f08dcd1656db1b82e3177425911091