Hello, I'm trying to understand Resources conceptu...
# ask-community
Hello, I'm trying to understand Resources conceptually. How are resources passed between dagster ops/assets? For example if i open a connection to an external service as part of the
of a ConfigurableResource, will that connection be preserved for all nodes in the asset materialization? or will each asset that accesses the resource re-open that connection? What's the best way to only open a connection once and have each asset use the same connection?
🤖 1
the examples in this post don't seem to use a persistent connection, but rather isolate the expensive logic into functions in the resource class...
@ben given your name is on the article, maybe you have some insight here?
resources are not shared between processes, they are re-initialized for every node in the graph. a lot of this seems to comes down to the fact that it's really difficult to share data between processes in python, particularly non-serializable data. it's a pretty common ask here to have a database resource which preserves a connection across nodes in your execution graph but as far as I know it's not currently supported
👍 1
I think this is kind of possible using the in-memory executor, but then you lose parallelization
how would you do this via the in-memory executor?
I think it pretty much just works by default... if you make a database connection resource, and configure it on ops in a job which uses the in-memory executor then it'll only instantiate the resource once, because there's only one process being used. I don't think there's anything particularly special you have to do (aside from configuring the job to use the in-memory executor)
cool, will give it a shot. Thanks!
🎉 1