The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hi community!
I was wondering if I can define in my config files to NOT pickle the Out() of all dagster operations in my jobs... I use a special class that cannot be pickled.
Thank you

Look into IO managers, that's exactly what they exist for 

Following on that <@U03JEQ2PRRV>, I have now written my own IO manager that can load and save my specific class types.
I do have a conceptual question though... every time I run an operation, does the IO manager always load and then save my files? Is there a way to keep the processed object in memory for multiple operations that each rely on the output of the previous one?
I am asking because the loading/saving step is much slower than any of my actual operations.
Thanks again

<@U03PTTR5RQS> it depends on what executor you're using.  The default executor is the `multiprocess_executor`, which uses a different process for each op/asset.  When they execute in different processes, then the values need to be stored outside of memory.

If you use the `in_process_executor`instead, then you can use the `mem_io_manager`, which stores outputs in memory.

Does that answer your question?