https://dagster.io/ logo
Title
s

Stefan Adelbert

03/08/2022, 4:45 AM
Fixed Fan-In with Order-Only Dependencies I'm trying to create a graph with data and order-based dependencies (see attached diagram).
op A
produces some data that ops in
graph C
depend on (data dependency).
op B1
,
op B2
,
op B3
, all need to complete before
graph C
should start (order-based dependency).
op B1
,
op B2
,
op B3
execution order is not important. Can anyone suggest a way to build this graph please?
For now I'll reorganise things to get it working, like in the attached diagram. But it would still be good to have a pattern for the original graph.
a

alex

03/14/2022, 2:31 PM
something like
@graph
def example():
    a = op_a()
    b1 = op_b1(a)
    b2 = op_b2(a)
    b3 = op_b3(a)
    graph_c(data=a, start_after=[b1, b2, b3])
should work, though if
graph_c
is a
@graph
, you will need to map the input through to all the inner ops that should block / receive data
s

Stefan Adelbert

03/20/2022, 5:12 AM
start_after
? What is this magick?
a

alex

03/21/2022, 3:16 PM
just me extrapolating an example, it assumes you map
start_after
in
graph_c
to some op that takes something like
ins={'start_after': In(Nothing)}
s

Stefan Adelbert

03/22/2022, 12:23 AM
@alex 😄 OK, so not magick then, just order-based dependency. I had tried what you suggest here, but ran into the problem you warn against, namely needing to map all the `start_after`™️ inputs to ops in the graph. This very quickly turned into a schmozzle and I was hoping for a cleaner and less fragile pattern.
a

alex

03/22/2022, 2:46 PM