https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Daniel Mosesson

07/21/2022, 10:55 AM
Is there a good way keep a handle to resources across a step boundary? I have some objects that use a resource that are returned from one step and then used in subsequent steps. On pickle, I delete the reference to the resource (as it is not pickleable) but I don't know to how to reconstruct it on the
__setstate__
side
The resource that I am taking about is basically a configured connection manager that knows about how to talk to an in house service (configuration params are where is the service, where are the needed certificates, etc). The object uses that for lazily loading more of it's properties from the service
j

jamie

07/21/2022, 1:26 PM
Hi @Daniel Mosesson I think i need a little more clarification on what you're trying to do. Are you trying to return a resource as part of an op's output? Or are you trying to ensure that a resource is accessible in multiple ops in your graph?
d

Daniel Mosesson

07/21/2022, 2:22 PM
I basically have a set of ops that do something like: • process data from set of sensors ◦ create some data assets from the data ◦ optionally update some settings on the sensors ◦ for failing sensors try some diagnostic steps ▪︎ update some sensor properties ▪︎ try rebooting the sensor ▪︎ create some internal tickets in the the "sensor service" The issue is that the the "sensor" service has no way to return a complete sensor object. All of the properties/data from that have to come from, and we had existing an existing python object that lazily loaded/modified data/properities as needed. What I have done is build a "connection" resource (fundamentally a
requests
Session
that knows how to connect to the sensor service (hostname, certificates, etc), and then add that as a static part of the class. Each of those `op`s then takes a list of sensors and then returns data, triggers side effects, etc. Between `op`s then, this static part of the class needs to get set (or move it from a static part of the class to an instance property so the
___setstate___
method can set it during the next op start). what I am now thinking about maybe I should have the list of sensors be a resource? Honestly not sure how to handle this situation
j

jamie

07/21/2022, 4:09 PM
ok i'm definitely still wrapping my head around this. maybe this is a dumb question but if you have a resource that can connect to the external sensor service, why do you need to pass that between ops? it seems to me that you should be able to attach the connection resource to each op (using
required_resource_keys
) and then each op would be able to interact with the sensor service. In this case i would see the data, sensor ids, etc as what's passed between ops. let me know if this misses some complexity in your setup!
5 Views