19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi again and thanks a ton for Dagster :smile: I'm ...
# announcementsw
wbonelli
07/29/2020, 4:39 PMHi again and thanks a ton for Dagster 😄 I'm wondering if there's an easy way to make dynamic pipelines (whose shape depends on some parameters) reconstructable, since
reconstructables
schrockn
07/29/2020, 6:43 PMUnfortunately there’s no straightforward way to do that (unless I am missing something @alex). But that would be interesting to do potentially as we could probably concoct a scheme to capture the arguments and serialize them, but that would be a substantial task.
Can you give exact context now how your are parameterizing pipeline creation and why?
schrockn
07/29/2020, 6:44 PMRe: the task you mentioned, I’m also in interested in hearing about your exact use case. We want to support some aspect of what I refer to as dynamic orchestration. However we also explictly don’t want to repurpose dagster as a map-reduce engine, which we feels should be handled by computational runtimes such as spark or dask.
w
wbonelli
07/29/2020, 7:01 PMAh ok, thanks
wbonelli
07/29/2020, 7:02 PMThat makes sense re: not wanting to reinvent map-reduce
👍 1
wbonelli
07/29/2020, 7:09 PMBasically the use case is dynamic parallelization, where solids are spawned to process some number of inputs which isn't known until runtime
wbonelli
07/29/2020, 7:13 PMthe "processing" is some arbitrary Singularity-containerized workflow in an HPC environment where Dagster deploys to Dask, which deploys to some resource manager like PBS/SLURM
wbonelli
07/29/2020, 7:15 PMit could just be done in Dask, but would like to use Dagster if possible 🙂
wbonelli
07/29/2020, 7:22 PMDo you have any subset of dynamic orchestration in mind that Dagster may end up supporting, out of curiosity?
s
schrockn
07/29/2020, 8:01 PMFire and forget parallel runs would be the obvious thing. The “map” but not the “reduce” component that require fine-grained coordination
🎉 1
👍 1
w
wbonelli
07/29/2020, 8:08 PMthat would be awesome. just the ability to "map" some arbitrary number of solids over a collection would cover our use case
s
schrockn
07/29/2020, 8:10 PMGreat to know