Hey everyone! I started looking at Dagster recently and fell in love with its amazing UI. Part of that feeling comes from the fact that we're currently using Luigi at my place of work and its UI is too minimalistic. However, I didn't find any comparisons of Dagster vs. Luigi over the internet so asking here. My question is: In Luigi, Targets -- data artifacts produced by Tasks (pipelines), are first-class citizens and each time you execute a pipeline, Luigi unfolds all its dependencies and for each of them decides whether it should be executed or not based on presence of output targets (a task is considered as completed if all its output targets exist). How does Dagster fit into this picture? If I have a parametrized dagster pipeline and some of its solids have already been executed and produced an output (in my case, it's hive partition in s3 written by pyspark), will dagster skip corresponding subgraphs of these finished solids and execute only what's needed? Thanks in advance, will be happy to hear any inputs on this!

Hey! Dagster has all the building blocks to support skipping downstream graphs if a “target output” already exists - but doesn’t do it in an out-of-the-box way at this time. For example you could do this manually today by using optional outputs in your solids https://dagster.readthedocs.io/en/0.6.4/sections/learn/tutorial/multiple_outputs.html#multiple-and-conditional-outputs

Thanks for your answer! How does dagster decide which donwstream graphs it should skip and which not? Luigi does it by checking if a task is complete, and task is considered complete iff its output(s) exist in the filesystem. And assumption is that task is idempotent, deterministic and it's output is fully determined by task name and parameters' values passed in. Does dagster do something similar under the hood?

How does dagster decide which donwstream graphs it should skip and which not?

So we do not have any system wide mode (yet) like the one you described for Luigi - but you can accomplish this manually by not yielding an optional output.

@solid
def example():
    expected_path = calculate()
    if exists(expected_path):
        return # exit early - downstream work skipped
    output = do_work()
    write(output, expected_path)
    yield Output(...)

Interesting. Just to reiterate and make sure I'm on the same page: am I right that if I have two different

@pipeline

-s which turn out to execute the same

@solid

with the same parameters, then I shouldn't expect that solid's output will be reused for a pipeline that runs last. So basically that solid will be executed twice despite it does exact same calculations (since input is the same, as well as rest of parameters). Is this correct? Thanks!

if I have two different

@pipeline

-s which turn out to execute the same

@solid

with the same parameters, then I shouldn’t expect that solid’s output will be reused for a pipeline that runs last

Correct - we don’t make global assumptions about idempotentcy so we don’t do this. In the future we will add an “on switch” to enable this type of behavior - but in the meantime you can implement it in user space.

Very cool, thanks for your detailed answers!

@alex hi, is it still the best way to perform skipping?