Hi I was wondering how would you advise in solving the follo

Question

Hi I was wondering how would you advise in solving the following issue I have a pipeline that takes certain input of data with a long sql query and does some heavy calculations and reporting on top This pipeline is schedule to run every day We now have the requirement on running the same process on different subsets of the data that may be overlapping ideally we would like this to be dynamic like we have a tag that marks what data corresponds to which partitions and gets redirected to the pipeline but only for that particular subset What mechanism could allow us to implement this clearly in dagster one thing I would rather not do is to modify the current pipeline to put `for`s inside every solid to execute each subset of the data as the execution is quite heavy and also because a failure in a subset would damage the state for all subsets

alex · Answer

So one path until we have proper map fan out support is to use skippable outputs So you would code the pipeline to support having to compute the full breadth of data then in some controlling solid you would `yield Output` for each thing that should be computed in the current run and then any down stream solids on the skipped `Output`s would just also skip

alex · Answer

this assumes you know the full set of possibilities at pipeline generation time

Cris · Answer

I see this would still be a single pipeline right

alex · Answer

```def example solid output defs= OutputDefinition name= foo is required=False OutputDefinition name= bar is required=False OutputDefinition name= biz is required=False config schema= what to execute str def control node context for x in context solid config what to execute yield Output value=something output name=x pipeline def pipeline outputs = control node compute stuff alias compute foo outputs foo compute stuff alias compute bar outputs bar compute stuff alias compute biz <http outputs biz|outputs biz> ```

Cris · Answer

And I guess the most dynamic we can get is in the solids instantiation right Like for subset in subsets create solid subset

Cris · Answer

Do you know a good way to abstract a good chunk of the solids this single pipeline is a mess haha

alex · Answer

have you started using ` composite solid` yet

alex · Answer

alex · Answer

The other dimension of cleverness you can toy with is the `should execute` function on `ScheduleDefinition` Here is an example where we have a fast ticking schedule that uses should execute to progressively do a backfill <https github com dagster io dagster blob master python modules dagster test dagster test toys schedules py L9 L87>

Cris · Answer

ooooh okok

Cris · Answer

Lemme see these examples then Thank you very much

Cris · Answer

< alex> in the example you have with the control node i though outputs should be a tuple is there a way to access these output elements by name like `output foo `

Cris · Answer

Ended up doing something like this ```DATA SUBSETS = subset 1 subset 2 subset 3 pipeline def heavy models pipeline outputs a dict with the outputs specified in the order of DATA SUBSETS elements subsets ouputs = data subset dispatcher subsets dict = subset name output for subset name output in zip DATA SUBSETS subsets ouputs for subset name in subsets dict single dataset heavy model alias subset name subsets dict subset name ``` It seems prety hacky haha so lemme know if you have a cleaner solution

Cris · Answer

This is the dispatcher ``` solid output defs= OutputDefinition name=n is required=False for n in DATA SUBSETS config schema= data subsets Field str description= list names of subdatasets to process at runtime def data subset dispatcher context data subsets can be read from DB for initialization instead of config However all the subsets in OutputDef should be fixed beforehand for subset name in context solid config subsets to process yield Output value=subset name output name=subset name ```

alex · Answer

its a `namedtuple` so you should be able to call ` asdict ` on it