https://dagster.io/ logo
t

Taylor

05/29/2019, 1:11 AM
I'm curious how dagster represents dags internally. Classical dags only have a notion of a single "input/output" for a given node and don't have dagster's concepts of inputs and outputs. If you are using a representation of this in a dag form, I'd love to understand how the multiple I/O is represented.
a

alex

05/29/2019, 3:03 PM
the data structure we use looks like
Dict[InputHandle, List[OutputHandle]]
where a "handle" is a tuple of node name and input/output name. Here is where we build up one for `Solid`s https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/definitions/dependency.py#L231 and here for `ExecutionStep`s https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/execution/plan/plan.py#L86
t

Taylor

05/29/2019, 3:33 PM
Awesome info - thanks @alex. For background I've prototyping developer tooling to support arbitrary dag development which I have working well on dbt dags, but not yet on dagster dags.