clay
04/25/2023, 10:14 PMclay
04/25/2023, 10:14 PMALTER SESSION SET query_tag = "whatever"
clay
04/26/2023, 1:17 PMjamie
04/27/2023, 8:46 PMjamie
05/05/2023, 4:29 PMSnowflakeResource
follows the new Pythonic Config and Resources system. The existing snowflake_resource
will continue to be supported. This SnowflakeResource works slightly differently than the snowflake_resource
. Using the new SnowflakeResource
you create a SnowflakeConnection
(snowflake docs) and use that to run queries
from dagster import asset
from dagster_snowflake import SnowflakeResource
@asset
def my_snowflake_asset(snowflake: SnowflakeResource):
with snowflake.get_connection() as conn:
return (
conn.cursor()
.execute(
"SELECT * FROM IRIS_DATASET WHERE 'petal_length_cm' < 1 AND"
" 'petal_width_cm' < 1"
)
.fetch_pandas_all()
)
Also, if you haven’t yet taken a look at the GitHub discussion about handling timestamp data, please let us know your thoughts!Dan Meyer
05/08/2023, 12:49 AMjamie
05/12/2023, 2:34 PMbase64
encode your key and the resource and I/O manager will decode the key and use it for authentication. If you are already successfully supplying a private key, then your code is still supported and no changes need to be made.Joe
05/12/2023, 6:35 PMSebastian Charrier
05/15/2023, 12:10 AMclay
05/16/2023, 4:10 PMSimrun Basuita
05/17/2023, 10:32 AMsnowflake_pandas_io_manager
to only load part of a table for a downstream asset? i.e. add a WHERE clauseSimrun Basuita
05/17/2023, 1:47 PMCREATE TABLE ....
), how can I get downstream assets to load that using snowflake_pandas_io_manager
? i.e. how to declare that the asset lives in snowflake?Rohan Meringenti
05/25/2023, 4:02 PMsnowflake_pandas_io_manager
as the docs say and time partitions to do this, but it looks like an @asset
always recreates the table. Was wondering what the suggested route was to go about updating a table on a daily basis in snowflake?Stephen Bailey
05/26/2023, 11:26 AMfoo1
from here, but it failed, then i expected to run foo2
like here but it failed. Then i tried foo3
from here, and that worked for me.
The pattern I'd prefer to encourage others to use is the first -- is that supported?
from dagster import Definitions, asset, OpExecutionContext
from dagster_snowflake import SnowflakeResource
@asset
def foo(context: OpExecutionContext, snowflake: SnowflakeResource):
result = snowflake.execute_query("select 1")
<http://context.log.info|context.log.info>(result)
@asset
def foo2(context: OpExecutionContext, snowflake: SnowflakeResource):
with snowflake.get_connection() as conn:
result = conn.execute_query("select 1")
<http://context.log.info|context.log.info>(result)
@asset
def foo3(context: OpExecutionContext, snowflake: SnowflakeResource):
with snowflake.get_connection() as conn:
result = conn.cursor().execute("select 1")
<http://context.log.info|context.log.info>(result)
defs = Definitions(
assets=[foo, foo2, foo3],
resources={
"snowflake": SnowflakeResource(
account="whatnot-prod",
user=SNOWFLAKE_USER,
password=SNOWFLAKE_PASSWORD,
role="STAGE_ADMIN_ROLE",
warehouse="STAGE_LOAD_WH",
database="STAGE",
)
}
)
Matt Clarke
05/31/2023, 12:30 PMMatt Clarke
06/05/2023, 5:42 PMAllPartitionMapping
, but I think that would get confused with backfillsBennie Regenold
06/08/2023, 2:41 PMMoi Stern
06/12/2023, 3:02 PMSnowflakeConnection
class for a few jobs - trying to query some data and then do stuff with it. I've been using the execute_query
method to extract the data I need from the warehouse. It seems like all of the sudden the method is no longer recognized:
AttributeError: 'SnowflakeConnection' object has no attribute 'execute_query'
I'm curious as to why this might've happened. Not sure if its something we did on our side. Or maybe it has to do with some update? Not sure. Any help would be very appreciatedStephen Bailey
06/15/2023, 5:06 PMStephen Bailey
06/15/2023, 5:21 PMSon Do
06/27/2023, 9:08 PMSimrun Basuita
07/05/2023, 8:22 AMautocommit=False
argument to snowflake.connector.connect
(https://docs.snowflake.com/developer-guide/python-connector/python-connector-example#using-context-manager-to-connect-and-control-transactions).
How can I get a Snowflake connection with autocommit=False
from SnowflakeResource.get_connection()
?Joel Olazagasti
07/25/2023, 9:43 PMNone
on prod (so that it infers the schema from the asset keys like normal)? We'd like to avoid giving our assets variable key names per enviroment, so our Asset graph correctly reflects the state it would be in prod. We use dagster-dbt so we aren't in full control of how the asset dependencies are managed. We've tried using a variable configuration on a SnowflakePandasIoManager
but when we're in a dev environment and try to override the asset key-defined schema named, we get an error for trying to define the schema in multiple places at onceSterling Paramore
08/11/2023, 4:28 PMMatt Clarke
08/16/2023, 10:39 PMDagster Jarred
08/22/2023, 7:44 PMHarkanwal Singh
08/29/2023, 3:18 AMBenjamin Faught
09/08/2023, 4:16 PMJason
09/08/2023, 5:26 PMsnowflake_resource
to the Pythonic SnowflakeResource
and I used to use something like snowflake.execute_query(...)
but now it looks like there is no execute_query
method for the Pythonic version and I now have to get a connection and query the db directly. Is this correct?
def get_one(snowflake_resource: SnowflakeResource):
with snowflake_resource.get_connection() as conn:
# conn is a snowflake.connector.Connection object
conn.cursor().execute("SELECT 1")
And to follow up that - is the expectation/migration then to instead of using execute_query(...use_pandas_result=true)
we should use cursor.fetch_pandas_all()
? Just wanted to make sure I'm not missing some documented/recommended approach. ThanksChris Graybeal
09/15/2023, 6:34 PM