The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hello!  Is it possible for the ops spawned from a DynamicOutput to run as they are yielded, or does the DynamicOutput have to finish processing before all parallel tasks can execute?

To clarify, using the process directory example from the docs.  Is it possible to use the DynamicOutput to run an op against each yielded file as soon as it is yielded instead of waiting for the entire directory to be crawled?

unfortunately no - the downstream ops will wait until all upstream deps have finished executing.

That's unfortunate.  Are there any suggested work arounds for accomplishing that?

hmm. One workaround is instead of using dynamic outputs, you actually just launch off entirely separate runs

so instead of yielding a dynamic output for each file in the directory, you send a run request