Salvador Ribolzi06/03/2022, 6:50 PM
, is there anywhere I can read more on this to better understand how and when resources are instantiated? This is a simple @ job with a couple @ op and the single @ resource in the same file
Depending on your executor, resources may be instantiated and cleaned up more than once in a job execution.
owen06/03/2022, 8:12 PM
, which will run each op in its own separate process. This means that ops will each have to instantiate their own separate copies of a resource when their process is initialized (as the resource object can't be shared across the process boundary). For simple local use cases, you can swap the execution mode such that all ops run in the same process: https://docs.dagster.io/concepts/ops-jobs-graphs/job-execution#default-job-executor
Zach06/03/2022, 8:19 PM