I m currently moving my research PyTorch code to Dagster The

Question

I m currently moving my research PyTorch code to Dagster The training is failing with the following error ```AttributeError Can t pickle local object SeqDataset make collate fn lt locals gt collate fn ``` This is happening inside the `torch DataLoader` I m using multiprocessing and `num workers` gt 0 But this happens even with `num workers=0` This did not happen outside of Dagster so I assume it has something to do with Dagster s `Multiprocess Executor` Sadly my understanding of multiprocessing is not the best so I m stuck here I was unable to google anything relevant Would appreciate any help The last lines of the error ```File usr lib python3 10 multiprocessing process py line 121 in start self popen = self Popen self File usr lib python3 10 multiprocessing context py line 224 in Popen return default context get context Process Popen process obj File usr lib python3 10 multiprocessing context py line 288 in Popen return Popen process obj File usr lib python3 10 multiprocessing popen spawn posix py line 32 in init super init process obj File usr lib python3 10 multiprocessing popen fork py line 19 in init self launch process obj File usr lib python3 10 multiprocessing popen spawn posix py line 47 in launch reduction dump process obj fp File usr lib python3 10 multiprocessing reduction py line 60 in dump ForkingPickler file protocol dump obj AttributeError Can t pickle local object SeqDataset make collate fn lt locals gt collate fn ```

Daniel Gafni · Answer

< owen> < sandy> maybe one of you guys can help

yuhan · Answer

do you need multiprocess if not you can turn it off by configuring the run ```execution config in process null ```

Daniel Gafni · Answer

I probably don t need it locally and in production K8s pods it won t matter right

sandy · Answer

gt in production K8s pods it won t matter right Right Btw I suspect what s going on here is that the asset function is accessing something defined at a scope outside the asset function When the python multiprocessing library forks a new process it needs to bring that object into the new process so it tries to pickle is but that thing isn t pickle able

Daniel Gafni · Answer

Thanks Sandy Basically that s true I have a `make collate fn` method that has a `def collate fn` inside It s a closure for some variables I wonder why this isn t an issue for torch s multiprocessing I refactored the function and it s working now

sandy · Answer

it s also possible that the forkserver start method would help with this <https docs dagster io concepts ops jobs graphs job execution default job executor>