Dmitry Persiyanov11/19/2019, 12:40 PM
alex11/19/2019, 4:07 PM
Dmitry Persiyanov11/19/2019, 4:22 PM
alex11/19/2019, 4:32 PM
How does dagster decide which donwstream graphs it should skip and which not?So we do not have any system wide mode (yet) like the one you described for Luigi - but you can accomplish this manually by not yielding an optional output.
@solid def example(): expected_path = calculate() if exists(expected_path): return # exit early - downstream work skipped output = do_work() write(output, expected_path) yield Output(...)
Dmitry Persiyanov11/19/2019, 4:50 PM
-s which turn out to execute the same
with the same parameters, then I shouldn't expect that solid's output will be reused for a pipeline that runs last. So basically that solid will be executed twice despite it does exact same calculations (since input is the same, as well as rest of parameters). Is this correct? Thanks!
alex11/19/2019, 6:02 PM
if I have two differentCorrect - we don’t make global assumptions about idempotentcy so we don’t do this. In the future we will add an “on switch” to enable this type of behavior - but in the meantime you can implement it in user space.-s which turn out to execute the same
@pipelinewith the same parameters, then I shouldn’t expect that solid’s output will be reused for a pipeline that runs last
Dmitry Persiyanov11/19/2019, 8:31 PM
Ilya Tyutin05/04/2020, 7:05 AM
I can't find a way to implement such logic at the moment. I would love to encapsulate that logic in one solid for convenience 🤔 Any suggestions?
if os.path.exists(...): # read the df from disk else: # run composite solid to get the df