https://dagster.io/ logo
#integration-snowflake
Title
# integration-snowflake
b

Bennie Regenold

06/08/2023, 2:41 PM
hi! I'm brand new to Dagster and have been trying to get some Snowflake examples to work, but it seems like most the examples in the docs don't work. Is it just me? Examples in 🧵
I tried to copy/paste the code from this example and got the following error:
Copy code
dagster._core.errors.DagsterInvalidDefinitionError: resource with key 'snowflake' required by op 'small_petals' was not provided. Please provide a <class 'dagster._core.definitions.resource_definition.ResourceDefinition'> to key 'snowflake', or change the required key to one of the following keys which points to an <class 'dagster._core.definitions.resource_definition.ResourceDefinition'>: ['io_manager']
I also tried to copy/paste this example, but the Iris dataset no longer exists and I get a 404 error
What I'm trying to figure out is: how do I query Snowflake to return a subset of a table (i.e. query with a
where
clause), then use that output in a downstream process
j

jamie

06/08/2023, 3:46 PM
Hey @Bennie Regenold the first error
resource with key 'snowflake' required
is because a resource with key
snowflake
isn’t in your Definitions object. Can you double check that you have this part of the code snippet
Copy code
defs = Definitions(
    assets=[small_petals],
    resources={
        "snowflake": SnowflakeResource(
            account="abc1234.us-east-1",
            user=EnvVar("SNOWFLAKE_USER"),
            password=EnvVar("SNOWFLAKE_PASSWORD"),
            database="FLOWERS",
            schema="IRIS",
        )
    },
)
and how are you materializing the asset? that might be a factor here as well
thanks for notifying us of the missing iris dataset. I’ll put up a fix for that soon
b

Bennie Regenold

06/08/2023, 3:47 PM
hi Jamie, thanks for the response! I do have that as part of the code, what I'm trying is this exact code block.
to materialize this, I ran
dagit
to get a local instance running, then found the asset and clicked
Materialize
j

jamie

06/08/2023, 3:48 PM
hmmm ok that should link the resources in correctly. let me run it myself and seee
so i’m not getting that error when i execute that code snippet. can you triple check that the spelling of snowflake matches in the Definitions and as the parameter to
small_petals
?
basically this error happens if you have something like this
Copy code
@asset
def small_petals(snowflake: SnowflakeResource) -> pd.DataFrame:
    ...


defs = Definitions(
    assets=[small_petals],
    resources={
        "not_snowflake": SnowflakeResource(
            account="abc1234.us-east-1",
            user=EnvVar("SNOWFLAKE_USER"),
            password=EnvVar("SNOWFLAKE_PASSWORD"),
            database="FLOWERS",
            schema="IRIS",
        )
    },
)
where the parameter name doesn’t have a match in the definitions
b

Bennie Regenold

06/08/2023, 4:08 PM
that did the trick!
I think the issue is that I had two
defs = Definitions()
in my project
is that not allowed?
j

jamie

06/08/2023, 4:08 PM
here’s a link to the iris dataset https://archive.ics.uci.edu/dataset/53/iris you can download the zip and then set the
read_csv
method to the local
iris.data
file. I’m putting up a pr so that we self host this dataset from now on
ty spinny 1
you can have two Defs, but you have to do some other stuff to make it work. When you’re just starting out, you should just start with a single Definitions
b

Bennie Regenold

06/08/2023, 4:09 PM
ok cool, that makes sense
j

jamie

06/08/2023, 4:10 PM
you can just put everything in that Definitions object. so like if you’re putting several examples together, then you just mash it into one Definitions object
b

Bennie Regenold

06/08/2023, 4:10 PM
I think I can get by with just one for now. Thank you for the help! I had been trying different things for like 3 hours and was super stuck but now I think I'm good to go
j

jamie

06/08/2023, 4:10 PM
cool! def let us know if you run into other issues
5 Views