Hi All! Thanks for this amazing tool. I am working...
# announcements
Hi All! Thanks for this amazing tool. I am working on a proof of concept for my team where we want to dynamically create the graph and distribute the execution over a dask cluster. Does dagster-dask currently support the experimental DynamicOutput? My pipeline fails at the moment the dynamic part should be executed, but I am not sure whether this is an error from my side or that it is just not supported yet.
executor does not support
and I think that is unlikely to change. The
executor is unique - most of our executors have a process that drives orchestration. The dagster dask executor on the other hand submits the full execution graph to dask up front and lets dask drive the orchestration. With this set-up its unclear how to modify the graph in flight dynamically.
clear, thanks!