:wave: everyone! Trying to understand if my use-c...
# ask-community
d
👋 everyone! Trying to understand if my use-case can be solved with Dagster 🙏 I need to build a ETL pipeline. ETL pipeline needs to be generalistic. Based on the input some of the paths will should never execute and not be waited in the fan-in op later on. Each path should have a sequential execution of collector op and then transformation. Only then it should be get to fan-in where there would be post-calulaction based on all tranformations together. If this makes sense, have anyone knowledge if this can be done? Thanks 🤗
o
Hi @Daniel Kontorovskyi! It sounds like you could accomplish this with dynamic outputs. The first op could emit a dynamic output for each path you want to execute, then be mapped to some graph of transformations. at the end, you can collect all of the outputs into a single list, and pass that into a single post-calculation op.