https://dagster.io/ logo
Powered by
s

Steve Pletcher

03/18/2021, 3:28 PM
running into what appears to be a bug trying to test preconfiguration of a resource using 
StringSource
fields that draw from an environment variable. reproduction code in the thread.
Copy code
@dataclass
class Thing:
    foo: str


@resource({
    'foo': Field(StringSource),
})
def thing(init_context):
    return Thing(init_context.resource_config['foo'])


@configured(thing)
def configured_thing(_config):
    return {
        'foo': {'env': 'FOO_VAR'}
    }


resource_context = InitResourceContext(resource_config={}, resource_def=configured_thing)
configured_thing.resource_fn(resource_context)
"""
Traceback (most recent call last):
  File "test.py", line 29, in <module>
    configured_thing.resource_fn(resource_context)
  File "test.py", line 18, in thing
    return Thing(init_context.resource_config['foo'])
KeyError: 'foo'
"""
behavior is identical regardless of whether the environment variable is defined (or empty vs nonempty)
is this because 
StringSource
only works for solids/resources that take a single parameter?
c

chris

03/18/2021, 3:33 PM
hey Steve, so the reason this doesnt work is that InitResourceContext expects that all the config for a resource has already been resolved when you pass it in
as in, it expects that our config system has already resolved "foo" to be some value by the time 
InitResourceContext
is initialized. In general, we consider 
InitResourceContext
to be an internal API, but the 
build_resources
API that Alex mentioned earlier will handle this case
s

Steve Pletcher

03/18/2021, 3:35 PM
got it. so when dagster spins up the resource context during a pipeline run, it'll properly resolve the env vars?
c

chris

03/18/2021, 3:35 PM
that's correct
s

Steve Pletcher

03/18/2021, 3:35 PM
great. i'll see if i can adapt my unit tests to use the 
build_resources
api
thanks for the help!
c

chris

03/18/2021, 3:36 PM
Sure, let me know if you have any questions or run into any problems!
I do just want to clarify that the API is still being actively developed, so you may run into some breaking changes in the near future. If that is too risky, then you could always spin up a fake pipeline for the express purpose of instantiating resources.
s

schrockn

03/18/2021, 3:44 PM
@Steve Pletcher you can just call 
os.environ.get('FOO_VAR')
directly in the 
configured
function
s

Steve Pletcher

03/18/2021, 3:45 PM
yeah, that's what we had been doing in the past, i just figured it was a better practice to use dagster's built-in mechanisms for that
i'm fine with keeping up with an experimental api for unit tests, since this won't be run in production
👍 1
ah, when alex mentioned this was going to be "released" in the coming months, i assumed they meant 'released in its final form,' i didn't realize that 
build_resources
itself wasn't available in 0.10.9
looks like spinning up a pipeline is the way to go for now - moving us up to bleeding-edge dagster might indeed be too risky for the team
Powered by