Jason Myers
08/01/2023, 6:52 PM@graph()
def data_table_graph(table_name, overlapping_periods: List[str] = []):
echo_data(table_name)
echo_data(overlapping_periods)
table_df = read_table_file(table_file_name=table_name)
echo_data_frame(table_df)
@graph
def first_graph():
table_file_list = build_table_list()
table_file_list.map(lambda table_file: data_table_graph(table_file, overlapping_periods))
Today when I execute it, it executes build_table_list() -> echo_data(table1) -> echo_data(table2).... -> read_table_file(table1)... -> echo_data_frame(table1)....
Is there a way to get it to run each data_table_graph in series rather than each subop? So something like build_table_list() -> echo_data(table1) -> read_table_file(table1) -> echo_data_frame(table1) ->echo_data(table2) -> read_table_file(table2) -> echo_data_frame(table2)
Basically more like a for loop, we're moving from Luigi and confused about how to do this properly, we tried limiting concurrency already with a
execution_config = {
"config": {
"multiprocess": {
"max_concurrent": 1,
},
},
}
return RunConfig(ops=config_map, execution=execution_config)
owen
08/01/2023, 8:26 PMJason Myers
08/01/2023, 8:37 PM