19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#announcements
Title
t
Tobias Macey
09/06/2019, 2:09 PMI've got a colleague who is evaluating Dagster for some data manipulation in one of our applications, but he needs to be able to generate nested subtasks based on inputs to a given solid. I feel like that's a conversation that has happened on here previously, but a cursory search didn't turn it up. Is that a capability that Dagster currently possesses?
s
schrockn
09/06/2019, 2:34 PMwe currently don’t support dynamic step creation based on data flowing through the dag
what is the problem they are trying to solve using that proposed feature?
t
Tobias Macey
09/06/2019, 2:36 PMSo, my understanding of the situation is that we have an incoming data structure that needs to get mapped to a different model, and depending on what is inbound there will be different nested attributes/objects that need to get handled via nested tasks to perform that mapping which is where the dynamic creation would be needed.
They looked at the
maps
schrockn
09/06/2019, 2:37 PMif there are bounded number of branches
this can be handled my multiple outputs quite nicely
*by
if it is totally unbounded and dynamic, we don’t support that atm
t
Tobias Macey
09/06/2019, 2:39 PMSo, multiple outputs meaning that for each component/subobject that needs to get processed, that would be generated as an output which would then get picked up as an input to the necessary step based on the type of the data?
s
schrockn
09/06/2019, 2:39 PMyeah
if i am understanding you correctly
meaning a solid can output A, B, or C
and you can optionally fire those outputs
so if you only fire B only those solids downstream from it get executed
t
Tobias Macey
09/06/2019, 2:41 PMRoger that.
s
schrockn
09/06/2019, 2:41 PM👍🏻
would love to hear more context on the use case at some point!
t
Tobias Macey
09/06/2019, 2:41 PMI just asked for a sample data structure and pseudo code for what we are doing so I can better understand and articulate the actual situation in the event that it serves as useful input for your work.
s
schrockn
09/06/2019, 2:42 PMwe are generally being conservative with adding dynamic features like that since it is complicated
and has deep effects throughout the system
seeing what we can do without it
but i suspect at some point we will support dynamic behavior like that in some form
t
Tobias Macey
09/06/2019, 2:54 PMSo, this is the imperative code that my colleague has written for this particular case https://github.com/mitodl/open-discussions/compare/nl/integrate_micromasters_catalog
His description of the overall logic flow is:
api call to edx to get course-> course run data
[10:51 AM]
then api call to MM for program-> course data, which merges over the edx data
[10:51 AM]
I basically need the transformed data to inform the structure of the load portion of the ETL pipeline
[10:53 AM]
but if I'm doing ET portion in a pipeline already, there's no way i could see under dagster to append new solids based on outputs of solids that have run so far
[10:53 AM]
in order to generate the L portion
new messages
[10:53 AM]
the answer might be that ET is one pipeline and L is another
a
alex
09/06/2019, 3:13 PMYa so you could model the pipeline as the full set of possible solids to run (assuming can be known ahead of time) and use optional outputs to only "activate" the subset of them that should be run for that given invocation.
Another option you sort of alluded to would be to split the pipeline. You can't dynamically change a pipeline mid run, but you can dynamically generate pipelines ahead of time. Using the
@pipeline