https://dagster.io/ logo
Title
s

Sireesha Kuchimanchi

11/04/2022, 3:49 PM
I am working with dagster and snowflake. I have created etl pipeline with dagster and extracted data from a csv file. Now I am trying to populate the extracted data into respective snowflake tables. So with what extension I should save the file and where should I save it? And what is the command to run the snowflake file?
r

rex

11/04/2022, 3:54 PM
You’ll want to use the
snowflake_io_manager
for this. You can see the API docs for an example: https://docs.dagster.io/_apidocs/libraries/dagster-snowflake#dagster_snowflake.build_snowflake_io_manager By setting up this abstraction with your Snowflake credentials, and then loading your data as a pandas dataframe, we can save your data to specified schemas in your Snowflake data warehouse. For an explanation of IO managers, see https://docs.dagster.io/concepts/io-management/io-managers#io-managers.
s

Sireesha Kuchimanchi

11/04/2022, 4:18 PM
Yeah, I have read that. My doubt is where should I save the file and what is the command to execute that?
r

rex

11/04/2022, 4:32 PM
You won’t need to explicitly save the file. When your asset implementation (i.e. the contents of your python function) loads the csv file and returns the dataframe in its body, the IO manager implementation will then take over and decide how to store that dataframe. In the case of the Snowflake IO manager, it will store it in a specified Snowflake schema and table.