https://dagster.io/ logo
Title
k

Kirk Stennett

04/29/2021, 4:06 PM
Hey, I'm getting:
dagster.core.errors.DagsterUnknownResourceError: Unknown resource `custom_resource`. Specify `custom_resource` as a required resource on the compute / config function that accessed it.
I have the resource decorated in it's own file with
@resource
and it's loaded in the mode definition as:
resource_defs={ "custom_resource": custom_resource }
Am I missing something obvious with how the resource needs to be loaded?
Figured it out, couldn't find it in the docs for a while but it looks like you need to add
required_resource_keys
to each solid. I saw this in the modes and resources section but I figured it was just for the
@resource
decorator not
@solid
as well.
👍 1
s

sandy

04/29/2021, 8:21 PM
Thanks for reporting this @Kirk Stennett. @yuhan @chris it looks like the "Accessing Resources in Solids" heading that was present on the pre-0.11.0 site is no longer present on the post 0.11.0 site https://docs.dagster.io/concepts/modes-resources vs. https://legacy-docs.dagster.io/overview/modes-resources-presets/modes-resources
y

yuhan

04/29/2021, 8:26 PM
ah we should add it back
👍 1
s

Sebastian Napiorkowski

09/01/2021, 6:01 PM
Hey, I'm having this issue after adding a hook to a pipeline. Can I omit setting the required resource on each solid and do it on the pipeline level?
c

chris

09/01/2021, 6:07 PM
Are you having the issue with accessing the resources on your hook?
s

Sebastian Napiorkowski

09/01/2021, 6:08 PM
The resource that is used in the hook, for example "slack" in your docs fails with the exception. This gets called on failure and then the resources that "slack" uses cant be resolved. Although they are in the required resources of the "slack" resource.
@sentry_message_on_failure
@pipeline(mode_defs=[MODE_DEV])
def failing_pipeline():
...

@failure_hook(required_resource_keys={'sentry'})
def sentry_message_on_failure(context: HookContext):
...

@resource(required_resource_keys={"config_vars", "env_name"})
def sentry(context):
...

MODE_DEV = ModeDefinition(name="development", resource_defs={'sentry': sentry, "config_vars": config_vars, "env_name": env_name}
In
sentry
neither
config_vars
nor
env_name
can be found, depends in which order I access it first. Dagit is saying: failing_solid ENGINE_EVENT Starting initialization of resources [sentry, io_manager]. So it's not naming/resolving the required keys from the resource?
c

chris

09/01/2021, 6:29 PM
Can you send the actual error message?
s

Sebastian Napiorkowski

09/01/2021, 6:30 PM
dagster.core.errors.DagsterSubprocessError: During multiprocess execution errors occurred in child processes: In process 17: dagster.core.errors.DagsterUnknownResourceError: Unknown resource
env_name
. Specify
env_name
as a required resource on the compute / config function that accessed it.
c

chris

09/01/2021, 6:31 PM
Are you using
env_name
anywhere other than the body of
sentry
?
s

Sebastian Napiorkowski

09/01/2021, 6:31 PM
Yes
c

chris

09/01/2021, 6:31 PM
where-ever else you use
env_name
, it needs to be a required resource key as well
s

Sebastian Napiorkowski

09/01/2021, 6:32 PM
It is of course.
c

chris

09/01/2021, 6:32 PM
in the example you sent, it looks like
env_name
isn't a required resource key in your hook
Unless you're saying the error is happening in a place where it is being provided?
1
s

Sebastian Napiorkowski

09/01/2021, 6:33 PM
I'm not using the env_name in the hook, the hook only takes sentry
So what I see happening is: The raised Exception, comes to the hook, the hook resolves the first resource:
sentry
.
sentry
requires the two other resources (which are provided in the mode) but fails to resolve it and says they are unkown.
c

chris

09/01/2021, 6:35 PM
hm I see. So here's my guess to what's happening:
We initialize all the resources at once in a given process. Due to the resource init failure, we're likely throwing away all the resources (even those initialized), and not passing them forward to failure hook
Yea this is a tricky edge case I think you stumbled into here. Thanks for surfacing!
s

Sebastian Napiorkowski

09/01/2021, 6:38 PM
Is it? Resolving the whole requirement "graph" worked with solids.
c

chris

09/01/2021, 6:41 PM
Is the failure that's triggering the hook happening during solid execution or resource initialization? From the code snippet that you posted, I got the impression that it was happening during resource initialization.
If not during resource initialization, then yea this is likely just a small bug in our resolver when using hooks w/ resource to resource dependencies
s

Sebastian Napiorkowski

09/01/2021, 6:44 PM
I don't know what you exactly mean with resource initialization. It does happen after I run the pipeline in dagit. So it's not complaining that the resources are not set.
c

chris

09/01/2021, 6:47 PM
Okay gotcha. This is definitely bug with using hooks that have multi-layered resources dependencies (hook depends on resource which depends on resource). Looking at the core execution code, I can see that we don't resolve the transitive resource dependencies on the hook path.
This is a pretty easy fix, should be able to get it out for tomorrow's release
s

Sebastian Napiorkowski

09/01/2021, 6:49 PM
Wow, that's great. So, I've got some good and bad news 😄 Thank you a lot. Tomorrow a new release comes out? Then I don't have to switch to an experimental version, perfect.
c

chris

09/01/2021, 6:53 PM
Yep! we release every thursday
s

Sebastian Napiorkowski

09/01/2021, 6:53 PM
@chris Thank you! Have a good day. I will go off for today!👋
👍 1