Hi #dagster-support Team, I'm Manoj based out of Chennai, India. I have been using dagster for the last 4 to 5 months in building a customized orchestration solution for my company I work with. I should admit that dagster is one of the greatest frameworks in building/tying data applications so easily. I'm using multiprocessing support of dagster by building reconstructible pipelines across process boundaries and I'm surprised to know that build_reconstructable_job does only accept reconstructable_args with JSON serialized tuple. The problem I face now is I want to pass a DB session object (which contains a threaded lock embedded object) as one of the arguments to reconstructable_args. Is that possible ? Any help regarding this would be greatly appreciated.

Hi Manoj, glad you've enjoyed using dagster so far. Unfortunately all arguments to

build_reconstructable_job

must be serializable as Dagster reconstructs the job across the process boundary using file pointers and other serializable args. I would try passing the arguments required to create the DB session object to

reconstructable_args

, then reconstructing the object across the process boundary.

Hi @claire Thanks for your time in responding to my query. Sure, let me try that out, albeit not a simple task 🙂