Salvador Ribolzi
06/03/2022, 6:50 PMDepending on your executor, resources may be instantiated and cleaned up more than once in a job execution.
, is there anywhere I can read more on this to better understand how and when resources are instantiated?
This is a simple @ job with a couple @ op and the single @ resource in the same fileowen
06/03/2022, 8:12 PMmultiprocess_executor
, which will run each op in its own separate process. This means that ops will each have to instantiate their own separate copies of a resource when their process is initialized (as the resource object can't be shared across the process boundary). For simple local use cases, you can swap the execution mode such that all ops run in the same process: https://docs.dagster.io/concepts/ops-jobs-graphs/job-execution#default-job-executorZach
06/03/2022, 8:19 PM