john eipe05/16/2021, 8:00 AM
Now based off this metadata that resides in file or DB - is it possible to create a dynamic pipeline?
eg: task-group seq A 1 B 1 C 2 task seq task-group a1 1. A a2 1. A b1 1. B b2 2. B c3 1. C task-group is only a logical wrapper that helps to group and control the execution flow, maybe composite solids are a fit here tasks are essentially solids, say @solid def a1(): pass Execution flow: Task group A and B starts parallely (seq=1) and the task a1, a2 run parallel (seq=1) but b2 runs only after b1 is complete. c3 starts after tasks in Task group A and B are complete.
schrockn05/16/2021, 4:49 PM
john eipe05/16/2021, 5:03 PM
Megan Beckett10/06/2022, 11:43 AM
schrockn10/06/2022, 12:28 PM