Donald Leonhard-MacDonald07/14/2021, 8:30 AM
on solids with dynamic outputs seems to collect all results first, only then apply any downstream solids (preventing infinite event streams) is this the expected behaviour? We have a few concerns: The time to instantiate a workflow for a single sample infernce could be quite slow. (can we keep workflows alive for multiple runs?) The workflows are blocking so we need multiple workflows, (are celery worker the solution to this?) Do you have any suggestions, ideas how we should be thinking about this?
sandy07/14/2021, 9:10 PM
Behavior ofThat is the current expected behavior. In the future, we may enable downstream steps to get kicked off before the step with the dynamic output completes, but infinite streaming is not a primary use case we have in mind.on solids with dynamic outputs seems to collect all results first, only then apply any downstream solids (preventing infinite event streams) is this the expected behaviour?
Donald Leonhard-MacDonald07/15/2021, 6:49 AM
sandy07/15/2021, 7:59 PM
Donald Leonhard-MacDonald07/16/2021, 6:19 AM
sandy07/16/2021, 4:54 PM
Donald Leonhard-MacDonald07/20/2021, 8:01 AM
sandy07/20/2021, 3:10 PM
Donald Leonhard-MacDonald07/21/2021, 2:21 PM
sandy07/21/2021, 8:57 PM
Donald Leonhard-MacDonald07/28/2021, 11:22 AM