https://dagster.io/ logo
#integration-snowflake
Title
# integration-snowflake
s

Simrun Basuita

05/17/2023, 1:47 PM
Another question: if I have an asset which creates a table in snowflake using the snowflake connector (i.e.
CREATE TABLE ....
), how can I get downstream assets to load that using
snowflake_pandas_io_manager
? i.e. how to declare that the asset lives in snowflake?
j

jamie

05/17/2023, 2:07 PM
if you create the table in a way that the snowflake io manager understands, you should be able to just load it. For example, you’ll need to make sure that the database and schema are correctly configured, and the name of the asset will need to match the name of the table that’s created.
Copy code
@asset
def my_table():
    # pseudocode! not real snowflake methods
    snowflake.execute("CREATE TABLE my_db.my_schema.my_table")

@asset
def downstream(my_table: pd.DataFrame):
    # do stuff

defs = Definitions(
    assets=[my_table, downstream],
    resources={
         "io_manager": SnowflakePandasIOManager(database="my_db", schema="my_schema", account=..., ...)
    }
)
s

Simrun Basuita

05/17/2023, 3:22 PM
Interesting, thanks.
Ah, but what if downstream() is using a different io manager?
j

jamie

05/18/2023, 2:52 PM
ah like if the output of
downstream
needs to be stored as a blob somewhere? you can set the io manager on
my_table
to snowflake pandas, and the io manager on
downstream
to whatever you want it to be. The io manager attached to
my_table
will always be used to load
my_table
in a downstream asset
Copy code
@asset(
    io_manager_key="snowflake_pandas"
)
def my_table():
   ...

@asset(
    io_manager_key="s3"
)
def downstream(my_table):
    ...

defs = Definitions(
    assets=[my_table, downstream],
    resources={
         "snowflake_pandas": SnowflakePandasIOManager(database="my_db", schema="my_schema", account=..., ...)
         "s3": ...
    }
)
s

Simrun Basuita

05/18/2023, 3:46 PM
Gotcha. And then I just have no return value in
my_table
function, so the I/O manager never actually gets invoked when it's run.
j

jamie

05/18/2023, 4:59 PM
yep exactly. technically the io manager will be invoked, but it won’t do anything since we have an immediate return if the output is None
s

Simrun Basuita

05/18/2023, 7:32 PM
thanks
2 Views