https://dagster.io/ logo
#integration-snowflake
Title
# integration-snowflake
c

clay

02/14/2023, 6:24 PM
Ok, I think this is a Snowflake question. 🙂 Let's say this is an asset that returns a DataFrame and that the asset is called
my_asset
. I can do this and create a DataFrame without a problem, which I return and from which Dagster creates a table in Snowflake.
Copy code
@asset
def my_asset(context) -> pd.DataFrame:
    with context.resources.snowflake.get_connection() as conn:
            with closing(conn.cursor()) as cursor:
                q2 = "SELECT * FROM SANDBOX.MY_SCHEMA.MY_TABLE"
                cursor.execute(q2)
                df = cursor.fetch_pandas_all()
                return df
However, if the following step has
my_asset
as a dependency, like this...
Copy code
@asset
def my_next_asset(context, my_asset) -> pd.DataFrame:
    stuff...
I get an error:
Copy code
dagster._check.CheckError: SnowflakeIOManager does not have a handler for type 'typing.Any'. Has handlers for types '<class 'pandas.core.frame.DataFrame'>'. Please add <class 'pandas.core.frame.DataFrame'> type hints to your assets and ops.
Why is it that I can pull the data in to create a DataFrame but the SnowflakeIOManager cannot do it without hints?
j

jamie

02/14/2023, 7:30 PM
So the snowflake io manager can be set up to handle different types at once. for example you could make a snowflake io manager that can handle both pandas and pyspark dataframes. In that case we require the typing hints to know which type to load the input as. However, in the case when the io manager is only set up to handle one type, we should default to using that type if there are no hints. I’ve been mentally tracking this issue, but haven’t had a chance to fix it yet. if you want to open a github issue with your code sample and assign it to me we can officially track it
c

clay

02/14/2023, 7:47 PM
ok, thanks!
9 Views