https://dagster.io/ logo
Join Slack
Powered by
Hi! I’m using the exact code snippet from the offi...
# integration-snowflake
a
Hi! I’m using the exact code snippet from the official docs https://dagster.io/integrations/dagster-snowflake but it gives the following error:
Copy code
AttributeError: 'SnowflakeCursor' object has no attribute 'execute_query'
I see this was already raised in #ask-ai channel but no working answer was provided. Would appreciate any guidelines. Thanks!
m
There are two connector backends for the snowflake resource: 
sqlalchemy
, and 
snowflake-connector-python
. If I recall correctly, one of them uses 
conn.execute_query()
, whilst the other uses 
conn.cursor().execute_query()
https://github.com/dagster-io/dagster/commit/45058bea4f8fe49d548eb1853ac5989ed4d1b0ae This recent commit shows the 
snowflake-io-manager
no longer uses the 
sqlalchemy
backend, and as a result some 
.cursor()
calls had to be added to the middle of the call chain, so something has changed in this domain recently.
a
Thanks a lot for the quick reply. Yeah, I’m reading the source code https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-snowflake/dagster_snowflake/resources.py#L37 right now and it’s in line with what you’re saying. Not sure yet though how then I can make it work 🙂
m
In one of our observable source assets we have this:
Copy code
context.resources.snowflake.execute_query(QUERY, fetch_results=True)
Which works fine
I think if you want to use the 
execute_query
method you need to drop the 
.cursor()
from your command.
I think cursors have 
execute
rather than 
execute_query
Copy code
with self.get_connection() as conn:
            with conn.cursor() as cursor:
                cursor.execute(sql_query, parameters)
                    return cursor.fetchall()
In the backend, the execute query method is doing something along these lines
So if you want to use the cursor directly you can do so
But there isn't a lot of point versus just using the execute_query method
The primary advantage of using it manually is it gives you access to additional methods like 
fetch_arrow
or 
fetch_pandas_batches
The former being great for type consistency, the latter being good for handling larger than memory datasets
a
Thanks a lot. Dropping 
.cursor()
, i.e., using having just
Copy code
with snowflake.get_connection() as conn:
    return conn.execute_query(QUERY)
doesn’t help, getting then
Copy code
AttributeError: 'SnowflakeConnection' object has no attribute 'execute_query'
I also tried 
execute
method, i.e.,
Copy code
with snowflake.get_connection() as conn:
    conn.cursor().execute(QUERY)
but then I’m getting
Copy code
TypeError: cannot pickle '_thread.lock' object
Interestingly, when I add 
.fetch_pandas_all()
to the previous call, i.e.,
Copy code
with snowflake.get_connection() as conn:
    return conn.cursor().execute(QUERY).fetch_pandas_all()
then it works!
3 Views
Open in Slack
PreviousNext
Powered by