Hi community! I was wondering if I can define in m...
# ask-community
Hi community! I was wondering if I can define in my config files to NOT pickle the Out() of all dagster operations in my jobs... I use a special class that cannot be pickled. Thank you
dagster bot responded by community 1
Look into IO managers, that's exactly what they exist for
Thank you
Following on that @Daniel Gafni, I have now written my own IO manager that can load and save my specific class types. I do have a conceptual question though... every time I run an operation, does the IO manager always load and then save my files? Is there a way to keep the processed object in memory for multiple operations that each rely on the output of the previous one? I am asking because the loading/saving step is much slower than any of my actual operations. Thanks again
@Jordi it depends on what executor you're using. The default executor is the
, which uses a different process for each op/asset. When they execute in different processes, then the values need to be stored outside of memory. If you use the `in_process_executor`instead, then you can use the
, which stores outputs in memory. Does that answer your question?
Yes it does. Thank you!