Casper Weiss Bang
01/19/2023, 12:03 PM@graph()
def fetch(payload: EgressTaskPayload):
max_objversion = get_current_max_objversion(payload)
return fetch_dataframe(payload, max_objversion)
the payload says whether we want to do incremental load or full load. the whole get_current_max_objversion
doesn't have any meaning if payload has incremental=True. The Op, however, has to run and simply return None right now. what i'd like would be the ability to write:
@graph()
def fetch(payload: EgressTaskPayload):
if payload.incremtal:
max_objversion = get_current_max_objversion(payload)
return fetch_dataframe(payload, max_objversion)
else:
return fetch_dataframe(payload)
However i do know that the whole graph
thing does it's magic, and doesn't actually execute, but build a graph definition.chris
01/21/2023, 12:28 AMCasper Weiss Bang
01/23/2023, 7:50 AM