I have a resource that I was using to collect information across multiple ops for an in_process_executor job which works but when I switch to the multiprocess execution, it's unable to retain its state (seems like it's creating new instance of it per op it's used). Is there a good way of handling this?

each op is executed in its own process in multiprocess execution - so an in memory data structure will not work. You will have to use something like the file system or python

multiprocessing

library to communicate between processes