Hello I m attempting to do something more complex than the i dagster #ask-community

Hello! I'm attempting to do something more complex...

Umar Hussain

01/07/2023, 11:14 PM

Hello! I'm attempting to do something more complex than the initial tutorials with

branching graphs

I have two ops:

Copy code

@op{out={'output_1' : Out(is_required=False)})
def get_parents_dataframe():
    df = pd.read_sql('...')
    data = df.groupby('parent')['id'].agg(list)
    for item in data.items():
        if item[0] == 'output_type_1':
            yield Output(item, outname='output_1')

Copy code

@op
def parse_file_output_1(context,_input) -> pd.DataFrame:

    return do_awesome_transforms()

and a graph

Copy code

@graph
def my_first_branch():
     output_1 = get_parents_dataframe() # as i understand this can have n outputs?
     parse_file(output_1)

unfortunately this returns the following error after I use

AssetsDefinition.from_graph(my_first_graph)

at my module level

__init___.py

(I'm using the project scaffold structure from the docs)

dagster._check.CheckError: Invariant failed. Description: All leaf nodes within graph 'my_first_branch' must generate outputs which are mapped to outputs of the graph, and produce assets. The following leaf node(s) are non-asset producing ops: {'parse_file_output_1'}. This behavior is not currently supported because these ops are not required for the creation of the associated asset(s).

🤖 1

sandy

01/09/2023, 6:31 PM

@Umar Hussain I think including

return

at the beginning of the last line might fix this

Umar Hussain

01/09/2023, 6:39 PM

@sandy on the graph ?

sandy

01/09/2023, 6:48 PM

Copy code

@graph
def my_first_branch():
     output_1 = get_parents_dataframe() # as i understand this can have n outputs?
     return parse_file(output_1)

👀 1

4 Views

Open in Slack

Previous Next