Another question: if I have an asset which creates a table in snowflake using the snowflake connector (i.e.

CREATE TABLE ....

), how can I get downstream assets to load that using

snowflake_pandas_io_manager

? i.e. how to declare that the asset lives in snowflake?

if you create the table in a way that the snowflake io manager understands, you should be able to just load it. For example, you’ll need to make sure that the database and schema are correctly configured, and the name of the asset will need to match the name of the table that’s created.

@asset
def my_table():
    # pseudocode! not real snowflake methods
    snowflake.execute("CREATE TABLE my_db.my_schema.my_table")

@asset
def downstream(my_table: pd.DataFrame):
    # do stuff

defs = Definitions(
    assets=[my_table, downstream],
    resources={
         "io_manager": SnowflakePandasIOManager(database="my_db", schema="my_schema", account=..., ...)
    }
)

Interesting, thanks.

ah like if the output of

downstream

needs to be stored as a blob somewhere? you can set the io manager on

my_table

to snowflake pandas, and the io manager on

downstream

to whatever you want it to be. The io manager attached to

my_table

will always be used to load

my_table

in a downstream asset

@asset(
    io_manager_key="snowflake_pandas"
)
def my_table():
   ...

@asset(
    io_manager_key="s3"
)
def downstream(my_table):
    ...

defs = Definitions(
    assets=[my_table, downstream],
    resources={
         "snowflake_pandas": SnowflakePandasIOManager(database="my_db", schema="my_schema", account=..., ...)
         "s3": ...
    }
)

Gotcha. And then I just have no return value in

my_table

function, so the I/O manager never actually gets invoked when it's run.

yep exactly. technically the io manager will be invoked, but it won’t do anything since we have an immediate return if the output is None

thanks