https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Stephen Bailey

04/25/2023, 2:57 PM
is it possible to use
__post_init_post_parse__
in pydantic resources? my use case is that i want to construct a client property on the resource, but only want to supply the secrets. here's what i expect to be able to do:
Copy code
from dagster import asset, materialize, ConfigurableResource

class FooResource(ConfigurableResource):
    api_key = "foo"

    def __post_init_post_parse__(self):
        self.client = get_client(self.api_key)

@asset
def thing(context, foo: FooResource):
    foo.client.do_something()


result = materialize([thing], resources={"foo": FooResource(api_key="test")})
print(result)
v

Vinnie

04/25/2023, 3:35 PM
The spaghetti I’ve been cooking on my resources that works reasonably well is something like this:
Copy code
class UtilsS3IOManager(ConfigurableIOManager):
    class Config:
        extra = "allow"
        frozen = False

    bucket: str

    @property
    def s3(self):
        if not hasattr(self, "_s3"):
            self._s3 = construct_s3_client()    
        return self._s3
🍝 1
s

Stephen Bailey

04/25/2023, 4:34 PM
ah, that's great
oi, with the following
Copy code
class DatadogApiResource(ConfigurableResource):
    api_key: str
    app_key: str

    @property
    def client(self):
        if not hasattr(self, "_client"):
            self._client = self.get_client()
        return self._client
i get this:
Copy code
dagster._core.errors.DagsterInvalidInvocationError: 'DatadogApiResource' is a Pythonic resource and does not support manipulating undeclared attribute '_client' as it inherits from 'pydantic.BaseModel' without extra="allow". If trying to maintain state on this resource, consider building a separate, stateful client class, and provide a method on the resource to construct and return the stateful client.
v

Vinnie

04/25/2023, 5:05 PM
You need the
class Config
block with the
extra
and
frozen
params as I sent to make sure the Resources aren’t frozen
👀 1
that’s why I referred to it as spaghetti, it’s probably not what the Dagster team would recommend 😄
s

Stephen Bailey

04/25/2023, 6:04 PM
aah, i totally missed that
a

Andras Somi

04/25/2023, 6:08 PM
You can use PrivateAttr from pydantic for defining internal variables, looks slightly nicer tha hasattr. https://dagster.slack.com/archives/C01U954MEER/p1682319379488309?thread_ts=1682226946.013319&channel=C01U954MEER&message_ts=1682319379.488309
💡 1
At least you don't need a config class
d

Daniel Vetter

04/25/2023, 6:13 PM
just my two cents, using the functools.cached_property seems to work pretty well and then you don't have to modify the class yourself
👀 1
v

Vinnie

04/25/2023, 6:14 PM
@Andras Somi this is much nicer! Will change mine to follow this as well
👍 1
s

schrockn

04/25/2023, 7:11 PM
cc: @ben
b

ben

04/27/2023, 8:12 PM
Thanks all for your feedback - we’re considering changing our recommendations around state management in resources and providing corresponding lifecycle methods. An RFC for this change is up here - the PR description goes over the rationale + new patterns. If you have any input please leave it there!
🙌 1
8 Views