hi team i have an issue with resource generator try finally

Question

hi team i have an issue with resource generator try finally when i run it using steplauncher my code is ```from dagster import Array Field Int Noneable String Dict Nothing Permissive op In Out graph resource from dagster core definitions no step launcher import no step launcher from dagster core execution plan external step import local external step launcher import tempfile import os resource def my resource context fd path = tempfile mkstemp text=True suffix= txt with os fdopen fd w as f f write Hello world try yield path finally Deleting file os remove path op required resource keys= my launcher my resource def my op context assert os path exists context resources my resource with open context resources my resource as f f read graph def my graph my op my job 1 = my graph to job name= my job 1 resource defs= my launcher no step launcher my resource my resource my job 2 = my graph to job name= my job 2 resource defs= my launcher local external step launcher my resource my resource config= resources my launcher config scratch dir tmp my launcher ``` running my job 1 works fine but my job 2 fails on assert code from finally block was executed before op code is it a bug or am i doing something wrong dagster version 0 14 13 Thanks

Zach · Answer

I think the code from the finally block is being executed before the op code because resources get initialized prior to the op code being executed and I m guessing they re not handled as generators so initialization executes the finally block as well as the yield You could try returning a proper context manager as your resource and that might work Something like ```from dagster import Array Field Int Noneable String Dict Nothing Permissive op In Out graph resource from dagster core definitions no step launcher import no step launcher from dagster core execution plan external step import local external step launcher import tempfile import os from contextlib import contextmanager resource def my resource context contextmanager def file handler fd path = tempfile mkstemp text=True suffix= txt with os fdopen fd w as f f write Hello world try yield path finally Deleting file os remove path return file handler op required resource keys= my launcher my resource def my op context with context resources my resource as file path assert os path exists file path with open file path as f f read graph def my graph my op my job 1 = my graph to job name= my job 1 resource defs= my launcher no step launcher my resource my resource my job 2 = my graph to job name= my job 2 resource defs= my launcher local external step launcher my resource my resource config= resources my launcher config scratch dir tmp my launcher ```

prha · Answer

Yep this sounds right Here s a doc section that you might find helpful <https docs dagster io concepts resources context manager resources>

Yan · Answer

< prha> < Zach> code works fine but i want to have global resource like spark session not to create destroy for every step if i run it in inprocess executor like here but to make it work also with external launcher

prha · Answer

Hi Yan I m pretty sure that even when using a contextmanager we will only instantiate one resource instance while using the `inprocess executor` It will just make sure the finally block executes when all the steps complete

Yan · Answer

but for me ```from dagster import Array Field Int Noneable String Dict Nothing Permissive op In Out graph resource from dagster core definitions no step launcher import no step launcher from dagster core execution plan external step import local external step launcher import tempfile import os from contextlib import contextmanager resource contextmanager def my resource context fd path = tempfile mkstemp text=True suffix= txt with os fdopen fd w as f f write Hello world try f Yielding file path yield path finally f Deleting file path os remove path op required resource keys= my launcher my resource def my op context path = context resources my resource assert os path exists path op ins= parent In Nothing required resource keys= my launcher my resource def my op 2 context path = context resources my resource assert os path exists path graph def my graph my op 2 my op my job 1 = my graph to job name= my job 1 resource defs= my launcher no step launcher my resource my resource my job 2 = my graph to job name= my job 2 resource defs= my launcher local external step launcher my resource my resource config= resources my launcher config scratch dir tmp my launcher ``` creates removes resource for every op if i run using inprocess ```2022 05 04 03 01 56 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65171 RUN START Started execution of run for my job 1 2022 05 04 03 01 56 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65171 ENGINE EVENT Executing steps using multiprocess executor parent process pid 65171 2022 05 04 03 01 56 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65171 my op ENGINE EVENT Launching subprocess for my op 2022 05 04 03 01 57 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65176 my op ENGINE EVENT Starting initialization of resources io manager my launcher my resource 2022 05 04 03 01 57 +0300 dagster INFO resource my resource 2a907531 954a 4e04 a60c 5cea65c75b0c my op Yielding file var folders 6 d1krd57x1h55htxqv69ds6vh39cs2s T tmpzy4hk7d7 txt 2022 05 04 03 01 57 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65176 my op ENGINE EVENT Finished initialization of resources io manager my launcher my resource 2022 05 04 03 01 57 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65176 my op LOGS CAPTURED Started capturing logs for step my op 2022 05 04 03 01 57 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65176 my op STEP START Started execution of step my op 2022 05 04 03 01 57 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65176 my op STEP OUTPUT Yielded output result of type Any Type check passed 2022 05 04 03 01 57 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65176 my op HANDLED OUTPUT Handled output result using IO manager io manager 2022 05 04 03 01 57 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65176 my op STEP SUCCESS Finished execution of step my op in 14ms 2022 05 04 03 01 57 +0300 dagster INFO resource my resource 2a907531 954a 4e04 a60c 5cea65c75b0c my op Deleting file var folders 6 d1krd57x1h55htxqv69ds6vh39cs2s T tmpzy4hk7d7 txt 2022 05 04 03 01 58 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65171 my op 2 ENGINE EVENT Launching subprocess for my op 2 2022 05 04 03 01 59 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65185 my op 2 ENGINE EVENT Starting initialization of resources io manager my launcher my resource 2022 05 04 03 01 59 +0300 dagster INFO resource my resource 2a907531 954a 4e04 a60c 5cea65c75b0c my op 2 Yielding file var folders 6 d1krd57x1h55htxqv69ds6vh39cs2s T tmpwf9w62zi txt 2022 05 04 03 01 59 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65185 my op 2 ENGINE EVENT Finished initialization of resources io manager my launcher my resource 2022 05 04 03 01 59 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65185 my op 2 LOGS CAPTURED Started capturing logs for step my op 2 2022 05 04 03 01 59 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65185 my op 2 STEP START Started execution of step my op 2 2022 05 04 03 01 59 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65185 my op 2 STEP OUTPUT Yielded output result of type Any Type check passed 2022 05 04 03 01 59 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65185 my op 2 HANDLED OUTPUT Handled output result using IO manager io manager 2022 05 04 03 01 59 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65185 my op 2 STEP SUCCESS Finished execution of step my op 2 in 14ms 2022 05 04 03 01 59 +0300 dagster INFO resource my resource 2a907531 954a 4e04 a60c 5cea65c75b0c my op 2 Deleting file var folders 6 d1krd57x1h55htxqv69ds6vh39cs2s T tmpwf9w62zi txt 2022 05 04 03 01 59 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65171 ENGINE EVENT Multiprocess executor parent process exiting after 2 68s pid 65171 2022 05 04 03 01 59 +0300 dagster DEBUG my job 1 2a907531 954a 4e04 a60c 5cea65c75b0c 65171 RUN SUCCESS Finished execution of run for my job 1 ``` and it assert still fails if i run using external executor ```2022 05 04 03 02 48 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65200 RUN START Started execution of run for my job 2 2022 05 04 03 02 48 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65200 ENGINE EVENT Executing steps using multiprocess executor parent process pid 65200 2022 05 04 03 02 48 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65200 my op ENGINE EVENT Launching subprocess for my op 2022 05 04 03 02 49 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65205 my op ENGINE EVENT Starting initialization of resources io manager my launcher my resource 2022 05 04 03 02 49 +0300 dagster INFO resource my resource 731db262 3560 459f 853f 34b37bad0700 my op Yielding file var folders 6 d1krd57x1h55htxqv69ds6vh39cs2s T tmpqflsk560 txt 2022 05 04 03 02 49 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65205 my op ENGINE EVENT Finished initialization of resources io manager my launcher my resource 2022 05 04 03 02 49 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65205 my op LOGS CAPTURED Started capturing logs for step my op 2022 05 04 03 02 50 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65214 my op ENGINE EVENT Starting initialization of resources io manager my launcher my resource 2022 05 04 03 02 50 +0300 dagster INFO resource my resource 731db262 3560 459f 853f 34b37bad0700 my op Yielding file var folders 6 d1krd57x1h55htxqv69ds6vh39cs2s T tmpff5hf02w txt 2022 05 04 03 02 50 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65214 my op ENGINE EVENT Finished initialization of resources io manager my launcher my resource 2022 05 04 03 02 50 +0300 dagster INFO resource my resource 731db262 3560 459f 853f 34b37bad0700 my op Deleting file var folders 6 d1krd57x1h55htxqv69ds6vh39cs2s T tmpff5hf02w txt 2022 05 04 03 02 50 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65214 my op STEP START Started execution of step my op 2022 05 04 03 02 50 +0300 dagster ERROR my job 2 731db262 3560 459f 853f 34b37bad0700 65214 my op STEP FAILURE Execution of step my op failed dagster core errors DagsterExecutionStepExecutionError Error occurred while executing op my op AssertionError Stack Trace File tmp my env lib python3 7 site packages dagster core execution plan utils py line 47 in solid execution error boundary yield File tmp my env lib python3 7 site packages dagster utils init py line 405 in iterate with context next output = next iterator File tmp my env lib python3 7 site packages dagster core execution plan compute generator py line 65 in coerce solid compute fn to iterator result = fn context kwargs if context arg provided else fn kwargs File test py line 25 in my op assert os path exists path 2022 05 04 03 02 50 +0300 dagster ERROR my job 2 731db262 3560 459f 853f 34b37bad0700 my op 2 Dependencies for step my op 2 failed my op Not executing 2022 05 04 03 02 50 +0300 dagster INFO resource my resource 731db262 3560 459f 853f 34b37bad0700 my op Deleting file var folders 6 d1krd57x1h55htxqv69ds6vh39cs2s T tmpqflsk560 txt 2022 05 04 03 02 50 +0300 dagster DEBUG my job 2 731db262 3560 459f 853f 34b37bad0700 65200 ENGINE EVENT Multiprocess executor parent process exiting after 2 51s pid 65200 2022 05 04 03 02 50 +0300 dagster ERROR my job 2 731db262 3560 459f 853f 34b37bad0700 65200 RUN FAILURE Execution of run for my job 2 failed Steps failed my op Error Pipeline run 731db262 3560 459f 853f 34b37bad0700 resulted in failure ```