i think in most cases it will be important to have proper names for the sub-DAGs (or even for the nodes of the sub-DAGs, for intermediate outputs), but we should still think about ways of making this kind of thing better to write

i know nick has a concept of encapsulating sub-DAGs as solids and drilling down into them in the dagit view

Another thing I think we'll see is cases where we will want to be able to easily switch between executing parallel sub-DAGs. In my trivial example below, in local context I want to load data to postgres, in the cloud I want to load data to Redshift.

Right now our machinery for enabling this is a bit clunky. We could set a global flag in the context, but then need to rewrite each individual solid to switch on that flag -- this is especially tricky when you think about wanting to use library solids without too much customization, and when you think about diamond topologies where the alternative sub-DAGs are relatively complex. We could also make the solids themselves more complex, for instance replacing

load_to_redshift

and

load_to_postgres

with a highly configurable generic

load_to_database

.

@max yeah so i think the ideal is to be able to have generic solids where the business logic is just coding against sql alchemy

and the only thing that differs is what engine is new’ed up during context creaiton

*creation

that will work for running sql queries, but it gets trickier when you start to think about stuff like efficiently batching uploads from spark to redshift

where sqlalchemy is impractical, and you start to have weird flavor-specific requirements, like temporary folders in s3

the generic "get stuff from spark to x database" solid can be written, but it's going to have a lot of config

yeah i guess i don’t have a sense of how custom those operations are

let me write it in the one solid with config style, and we can feel it out

👍

i think this issue will resurface when we hit more complex DAGs, but maybe for those the right answer is "have two pipelines that reproduce a lot of the same structure and have different paths in particular sub-DAGs" rather than writing switching machinery

you can also reuse solids between pipelines

yep, even easier if we get somewhere on higher-order solids/encapsulation

in terms of the very similar dag structures

between the different data types (i assume that thing that varies)

do you think they should be consolidated somehow? i guess i’m saying if their core computation does something different then they need to be separate

so i'm talking about cases where the computation and the dependencies inside the sub-dag are identical, but the inputs are different and it would be useful for the outputs to be independently named so that downstream parts of the dag can grab them

oh i see

"map this dag over these inputs and spit out outputs with these names"

@max do you have mac or not mac? My zoom also works backwards, I assumed it's some magic that mac users are used to

@freiksenet i'm on a mac too

Rx is such a pile of garbage

like what the fuck is that shit even

is that a programming library or a fucking candy crush?

hahahaha

lol how did i miss this