https://dagster.io/ logo
Join Slack
Powered by
hi! I'm brand new to Dagster and have been trying ...
# integration-snowflake
b
hi! I'm brand new to Dagster and have been trying to get some Snowflake examples to work, but it seems like most the examples in the docs don't work. Is it just me? Examples in 🧵
I tried to copy/paste the code from this example and got the following error:
Copy code
dagster._core.errors.DagsterInvalidDefinitionError: resource with key 'snowflake' required by op 'small_petals' was not provided. Please provide a <class 'dagster._core.definitions.resource_definition.ResourceDefinition'> to key 'snowflake', or change the required key to one of the following keys which points to an <class 'dagster._core.definitions.resource_definition.ResourceDefinition'>: ['io_manager']
I also tried to copy/paste this example, but the Iris dataset no longer exists and I get a 404 error
What I'm trying to figure out is: how do I query Snowflake to return a subset of a table (i.e. query with a 
where
clause), then use that output in a downstream process
j
Hey @Bennie Regenold the first error 
resource with key 'snowflake' required
is because a resource with key 
snowflake
isn’t in your Definitions object. Can you double check that you have this part of the code snippet
Copy code
defs = Definitions(
    assets=[small_petals],
    resources={
        "snowflake": SnowflakeResource(
            account="abc1234.us-east-1",
            user=EnvVar("SNOWFLAKE_USER"),
            password=EnvVar("SNOWFLAKE_PASSWORD"),
            database="FLOWERS",
            schema="IRIS",
        )
    },
)
and how are you materializing the asset? that might be a factor here as well
thanks for notifying us of the missing iris dataset. I’ll put up a fix for that soon
b
hi Jamie, thanks for the response! I do have that as part of the code, what I'm trying is this exact code block.
to materialize this, I ran 
dagit
to get a local instance running, then found the asset and clicked 
Materialize
j
hmmm ok that should link the resources in correctly. let me run it myself and seee
so i’m not getting that error when i execute that code snippet. can you triple check that the spelling of snowflake matches in the Definitions and as the parameter to 
small_petals
?
basically this error happens if you have something like this
Copy code
@asset
def small_petals(snowflake: SnowflakeResource) -> pd.DataFrame:
    ...


defs = Definitions(
    assets=[small_petals],
    resources={
        "not_snowflake": SnowflakeResource(
            account="abc1234.us-east-1",
            user=EnvVar("SNOWFLAKE_USER"),
            password=EnvVar("SNOWFLAKE_PASSWORD"),
            database="FLOWERS",
            schema="IRIS",
        )
    },
)
where the parameter name doesn’t have a match in the definitions
b
that did the trick!
I think the issue is that I had two 
defs = Definitions()
in my project
is that not allowed?
j
here’s a link to the iris dataset https://archive.ics.uci.edu/dataset/53/iris you can download the zip and then set the 
read_csv
method to the local 
iris.data
file. I’m putting up a pr so that we self host this dataset from now on
ty spinny 1
you can have two Defs, but you have to do some other stuff to make it work. When you’re just starting out, you should just start with a single Definitions
b
ok cool, that makes sense
j
you can just put everything in that Definitions object. so like if you’re putting several examples together, then you just mash it into one Definitions object
b
I think I can get by with just one for now. Thank you for the help! I had been trying different things for like 3 hours and was super stuck but now I think I'm good to go
j
cool! def let us know if you run into other issues
5 Views
Open in Slack
PreviousNext
Powered by