https://dagster.io/ logo
#integration-snowflake
Title
# integration-snowflake
a

Andrzej Drzystek

02/13/2024, 9:18 AM
Hi! I’m using the exact code snippet from the official docs https://dagster.io/integrations/dagster-snowflake but it gives the following error:
Copy code
AttributeError: 'SnowflakeCursor' object has no attribute 'execute_query'
I see this was already raised in #ask-ai channel but no working answer was provided. Would appreciate any guidelines. Thanks!
m

Matt Clarke

02/13/2024, 9:28 AM
There are two connector backends for the snowflake resource:
sqlalchemy
, and
snowflake-connector-python
. If I recall correctly, one of them uses
conn.execute_query()
, whilst the other uses
conn.cursor().execute_query()
https://github.com/dagster-io/dagster/commit/45058bea4f8fe49d548eb1853ac5989ed4d1b0ae This recent commit shows the
snowflake-io-manager
no longer uses the
sqlalchemy
backend, and as a result some
.cursor()
calls had to be added to the middle of the call chain, so something has changed in this domain recently.
a

Andrzej Drzystek

02/13/2024, 9:45 AM
Thanks a lot for the quick reply. Yeah, I’m reading the source code https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-snowflake/dagster_snowflake/resources.py#L37 right now and it’s in line with what you’re saying. Not sure yet though how then I can make it work 🙂
m

Matt Clarke

02/13/2024, 10:29 AM
In one of our observable source assets we have this:
Copy code
context.resources.snowflake.execute_query(QUERY, fetch_results=True)
Which works fine
I think if you want to use the
execute_query
method you need to drop the
.cursor()
from your command.
I think cursors have
execute
rather than
execute_query
Copy code
with self.get_connection() as conn:
            with conn.cursor() as cursor:
                cursor.execute(sql_query, parameters)
                    return cursor.fetchall()
In the backend, the execute query method is doing something along these lines
So if you want to use the cursor directly you can do so
But there isn't a lot of point versus just using the execute_query method
The primary advantage of using it manually is it gives you access to additional methods like
fetch_arrow
or
fetch_pandas_batches
The former being great for type consistency, the latter being good for handling larger than memory datasets
a

Andrzej Drzystek

02/13/2024, 10:43 AM
Thanks a lot. Dropping
.cursor()
, i.e., using having just
Copy code
with snowflake.get_connection() as conn:
    return conn.execute_query(QUERY)
doesn’t help, getting then
Copy code
AttributeError: 'SnowflakeConnection' object has no attribute 'execute_query'
I also tried
execute
method, i.e.,
Copy code
with snowflake.get_connection() as conn:
    conn.cursor().execute(QUERY)
but then I’m getting
Copy code
TypeError: cannot pickle '_thread.lock' object
Interestingly, when I add
.fetch_pandas_all()
to the previous call, i.e.,
Copy code
with snowflake.get_connection() as conn:
    return conn.cursor().execute(QUERY).fetch_pandas_all()
then it works!
2 Views