https://dagster.io/ logo
#announcements
Title
# announcements
c

Chet Lemon

04/05/2021, 7:06 PM
hello, I'm wondering if anyone has tried to pickle a solid before or if maybe there's a better way to serialize these for context, we are trying to pickle
{Solid,CompositeSolid,Pipeline}ExecutionResult
objects and solids are contained as attributes in these simple example:
Copy code
import pickle
import dagster


@dagster.solid
def test(context):
    pass

with open('/home/chet/data/testpickle.pkl', 'w') as p:
    pickle.dump(test, p)
Copy code
---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-45-e900432760d1> in <module>
     13 
     14 with open('/home/chet/data/testpickle.pkl', 'w') as p:
---> 15     pickle.dump(test, p)
     16 
     17 

PicklingError: Can't pickle <function test at 0x7f9bf70ab9d8>: it's not the same object as __main__.test
👀 1
a

adamd

04/05/2021, 7:22 PM
Alternative followup: Is there a recommended method to serialize/deserialize execution result objects?
a

alex

04/05/2021, 7:43 PM
The way these result objects are currently set up makes them pretty infeasible to serialize. Can you talk more about the overall goal for what you are trying to accomplish?
c

Chet Lemon

04/05/2021, 7:59 PM
okay, got it We are currently distributing dagster pipelines with k8s jobs over a cluster and wanted to pickle the results and pull them down from a cloud storage bucket for verification in one of our CI pipelines We thought this could be a short-term solution to fix a bug we were having, but we have been following updates to the dagit's kubernetes runners and are excited to check that out. (currently on 0.9.15)
a

alex

04/05/2021, 8:33 PM
Ah interesting - our general model is that you would have all the executions use the same
DagsterInstance
then could just interact with the instance to investigate results. Assuming you know what information you want off of the result object, you could make your own serializable object from stuff fetched off of the result. The
event_list
property should be serializable for example.
c

Chet Lemon

04/07/2021, 4:10 PM
thank you for steering us in the right direction! We have this solved now by serializing certain parts of the result for the time being
dagsir 1