Hello, I'm trying to understand Resources conceptually. How are resources passed between dagster ops/assets? For example if i open a connection to an external service as part of the

__init__

of a ConfigurableResource, will that connection be preserved for all nodes in the asset materialization? or will each asset that accesses the resource re-open that connection? What's the best way to only open a connection once and have each asset use the same connection?

resources are not shared between processes, they are re-initialized for every node in the graph. a lot of this seems to comes down to the fact that it's really difficult to share data between processes in python, particularly non-serializable data. it's a pretty common ask here to have a database resource which preserves a connection across nodes in your execution graph but as far as I know it's not currently supported

how would you do this via the in-memory executor?

I think it pretty much just works by default... if you make a database connection resource, and configure it on ops in a job which uses the in-memory executor then it'll only instantiate the resource once, because there's only one process being used. I don't think there's anything particularly special you have to do (aside from configuring the job to use the in-memory executor)

cool, will give it a shot. Thanks!