https://dagster.io/ logo
#integration-snowflake
Title
# integration-snowflake
s

Stephen Bailey

05/26/2023, 11:26 AM
hey all - trying to use the snowflake resource and I can't make heads or tails of what patterns are supported. I expected to be able to run
foo1
from here, but it failed, then i expected to run
foo2
like here but it failed. Then i tried
foo3
from here, and that worked for me. The pattern I'd prefer to encourage others to use is the first -- is that supported?
Copy code
from dagster import Definitions, asset, OpExecutionContext
from dagster_snowflake import SnowflakeResource

@asset
def foo(context: OpExecutionContext, snowflake: SnowflakeResource):
    result = snowflake.execute_query("select 1")
    <http://context.log.info|context.log.info>(result)

@asset
def foo2(context: OpExecutionContext, snowflake: SnowflakeResource):
    with snowflake.get_connection() as conn:
        result = conn.execute_query("select 1")
    <http://context.log.info|context.log.info>(result)

@asset
def foo3(context: OpExecutionContext, snowflake: SnowflakeResource):
    with snowflake.get_connection() as conn:
        result = conn.cursor().execute("select 1")
    <http://context.log.info|context.log.info>(result)

defs = Definitions(
    assets=[foo, foo2, foo3],
    resources={
        "snowflake": SnowflakeResource(
            account="whatnot-prod",
            user=SNOWFLAKE_USER,
            password=SNOWFLAKE_PASSWORD,
            role="STAGE_ADMIN_ROLE",
            warehouse="STAGE_LOAD_WH",
            database="STAGE",
        )
    }
)
Throw in here that it's really not clear to me what the difference is between
SnowflakeResource
, which does not have a dedicated entry on the docs page, and the
SnowflakeConnection
, which is apparently what's loaded within the body of the op?
j

jamie

05/26/2023, 1:59 PM
ack SnowflakeResource should definitely be in the docs. That’s an oversight/bug as i’m pretty sure it was there at one point. apologies for that. So the difference in API here is because in the old style snowflake resource
snowflake_resource
we wrote our own SnowflakeConnection class with those helper functions. For the new pythonic snowflake resource
SnowflakeResource
we instead chose to just directly wrap snowflakes
SnowflakeConnector
class (an annoying and unfortunate naming collision)
we can definitely do a better job explaining this in the API docs
the reason we went snowflake’s Connection object is because we felt it offered more flexibility to users and reduced some maintenance on our part (ie keeping up with any changes in the snowflake api so we could correctly wrap them in our api). However, if there’s a need to incorporate more helper functions in the new resource, we can certainly consider it!
s

Stephen Bailey

05/26/2023, 2:41 PM
I can understand that perspective, but I think at least offering
execute_query
and
execute_queries
for parity (even without the
fetch_pandas_result
, would be nice. Otherwise there's almost no benefit to using a resource class except for config management. But, if that's the approach that's going to be taken with other adapters (basically no additional functionality on resources), then it makes sense.
j

jamie

05/26/2023, 3:50 PM
yeah i’ll bring it up with some other folks and we can chat about it
👍 1
14 Views