Hi, I am facing an issue with dynamic output. I want to perform an operation that reads a dataframe and dynamically yields a row as an output and then that row should now be used for other ops to perform some function.
So, what I need is that the later rows shouldn't be yielded by the first op until all the ops related to the earlier row has been completed.
Somewhat like the whole job should first run for 1st row and until its completed the new row shouldn't be yielded or like the ops for new row shouldn't start, as it is in a wait and call method.
12/01/2022, 4:29 PM
DynamicOutput yielded by your Ops will get mapped to an Op executed in parallel. If you want the rows to be processed sequentially, I’m not sure this design fits
12/01/2022, 4:30 PM
So is there some way to pass only one row at a time with some other design and the make sure all other rows wait for the earlier row to execute completely??
12/01/2022, 7:33 PM
What’s your use case? In production settings processing a single row per op is probably going to be really expensive in a production setting. I think you should probably consider containing the entire processing of the data frame to a single op