The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

I like that conditional branching is possible, however the inability to have an actual if-statement on the graph level is sortof a pain to me. It might be difficult to solve, but readability is hurt by it IMO

```@graph()
def fetch(payload: EgressTaskPayload):
    max_objversion = get_current_max_objversion(payload)
    return fetch_dataframe(payload, max_objversion)```
the payload says whether we want to do incremental load or full load. the whole `get_current_max_objversion`  doesn't have any meaning if payload has incremental=True. The Op, however, has to run and simply return None right now. what i'd like would be the ability to write:
```@graph()
def fetch(payload: EgressTaskPayload):
    if payload.incremtal:
        max_objversion = get_current_max_objversion(payload)
        return fetch_dataframe(payload, max_objversion)
    else:
        return fetch_dataframe(payload)```
However i do know that the whole `graph` thing does it's magic, and doesn't actually execute, but build a graph definition.

yea it’s definitely a difficult problem, for the exact reason that you identify (graphs have magic dsl that powers them)

It would require a big lift I think to get it to work

I get that. Maybe someone has a eureka moment someday with a feature like that. i mean it is doable as is, and maybe I just need to create simpler graphs. Another case is that i don't want to trigger a "push" of data if the pull is empty. But again i can simply shortcurcuit the op to finish instantly if no data is fetched. so it isn't a major issue.