like subpipelines etc?

ya

nod

so in terms of why things need to be broken down

i build the url and header as per the library

in allscripts.py

is instructive

but i think your overall assessment of the pipelines being weird is accurate

jacob isn’t too disciplined

so they were structured in pretty arbitrary ways

i think also this might have been the first code generation system he

he’s ever built

I wasn't trying to criticize tbh, more like evaluating which use cases are valid (and thus visualization is fucked up) and which are just shouldn't be written this way

right

i would say

the ability_nightly one is the reasonable one

cjr_fileload is currently just a poorly structured dag

hey want your thoughts on this. this is from a convo in the superconductive slack

“@jacob.vi the cjr thing is interesting. this could be handled by the framework, but I’m not sure it is a good idea meaning that I don’t think it’s really a bad thing to add unnecessary edges or redundant edges so we could have 1) core dagster detect redundant edges and remove them (e.g. if C depends on B and A, and B depends on A, the C to A dependency could be removed without consequence) however that could be super confusing in certain cases maybe we should just instead eliminate on the display time in dagit *on display time so ya three options 1) do nothing. (current state) 2) fix deps in the core dagster pipeline definition 3) rely on tooling to optionally do redundant edge checking”

*this is a message from me

i’m interested in what you thin

I'm kina lacking context hre

kinda

here

this PR

the CJR pipeline

had tons of edges

that strictly weren’t necessary

(e.g. like every node depended on node one)