Stephen Bailey
05/26/2023, 11:26 AMfoo1
from here, but it failed, then i expected to run foo2
like here but it failed. Then i tried foo3
from here, and that worked for me.
The pattern I'd prefer to encourage others to use is the first -- is that supported?
from dagster import Definitions, asset, OpExecutionContext
from dagster_snowflake import SnowflakeResource
@asset
def foo(context: OpExecutionContext, snowflake: SnowflakeResource):
result = snowflake.execute_query("select 1")
<http://context.log.info|context.log.info>(result)
@asset
def foo2(context: OpExecutionContext, snowflake: SnowflakeResource):
with snowflake.get_connection() as conn:
result = conn.execute_query("select 1")
<http://context.log.info|context.log.info>(result)
@asset
def foo3(context: OpExecutionContext, snowflake: SnowflakeResource):
with snowflake.get_connection() as conn:
result = conn.cursor().execute("select 1")
<http://context.log.info|context.log.info>(result)
defs = Definitions(
assets=[foo, foo2, foo3],
resources={
"snowflake": SnowflakeResource(
account="whatnot-prod",
user=SNOWFLAKE_USER,
password=SNOWFLAKE_PASSWORD,
role="STAGE_ADMIN_ROLE",
warehouse="STAGE_LOAD_WH",
database="STAGE",
)
}
)
Stephen Bailey
05/26/2023, 11:28 AMSnowflakeResource
, which does not have a dedicated entry on the docs page, and the SnowflakeConnection
, which is apparently what's loaded within the body of the op?jamie
05/26/2023, 1:59 PMsnowflake_resource
we wrote our own SnowflakeConnection class with those helper functions. For the new pythonic snowflake resource SnowflakeResource
we instead chose to just directly wrap snowflakes SnowflakeConnector
class (an annoying and unfortunate naming collision)jamie
05/26/2023, 1:59 PMjamie
05/26/2023, 2:12 PMStephen Bailey
05/26/2023, 2:41 PMexecute_query
and execute_queries
for parity (even without the fetch_pandas_result
, would be nice. Otherwise there's almost no benefit to using a resource class except for config management. But, if that's the approach that's going to be taken with other adapters (basically no additional functionality on resources), then it makes sense.jamie
05/26/2023, 3:50 PM