Hi
I want to use memoization for my pipeline
My versioned io manager has dependency on pyspark resource
And i see that for each solid in the pipeline new pyspark resource is being created.
Is there any way to avoid this?
Thanks!
a
alex
06/15/2021, 2:17 PM
i assume this is multiprocess execution? Each process needs its own instance of the resource