following up on this a little bit, I think the con...
# announcements
d
following up on this a little bit, I think the consensus here was to use the
``Nothing``
input/output type to model dependencies between solids that don't rely on input/output. This seems to work well most of the time, but I'm getting stuck on how to leverage this for reusable solids that have a dynamic amount of upstream and/or downstream dependencies. An example of this is a
``dbt_run``
solid we have. This
``dbt_run``
solid can theoretically be dependent on 1 to n number of upstream solids. Currently, it feels like the only way to model this in Dagster is to create the solid with n number of
``Nothing``
input definitions. This doesn't feel intuitive to me so I'm wondering if there is something I'm missing here.
m
hmm, not that it's the most beautiful, but what about an input of type
``List[Nothing]``
a
you can fan-in on dependencies using a list if in a composition function or
``MultiDependencyDefinition``
if constructing directly
m
yeah exactly
though it's worth asking whether we should sugar the entire
``Nothing``
machinery by providing composition function syntax that lets you just have solids depend on each other
``solid_b.after(...)``
or
``solid_b().depending_on(...)``
or something
d
m
or even (yuck) the bitshift operators
d
for all of Airflow's faults (and there are many) being able to arbitrarily define dependencies between tasks regardless of input/output was nice
m
@dwall the big question is what kind of syntax is readable and useful here
i'd love to know how you wish you could write these dependencies down
d
I'm certain Im missing a ton of nuance here, but it feels as if a
``Nothing``
type input definition shouldn't really care about the number of Nothings it receives
a
in the interim this should work:
Copy code
``````x = solid_a()
y = solid_b()
z = solid_c()
dbt_run(on_complete=[x, y, z])``````
m
@alex are all
``Nothing``
inputs multi-dependencies? like
``List[Nothing]``
is
``Nothing``
?
a
Nothing type input definition shouldn’t really care about the number of Nothings it receives
If this isn’t how it works today it probably should
d
I can def check again. I think I remember dagster yelling at me
a
let me see if i can find a test - I think i made it work this way
if not its worth changing
d
@alex let me know what you find. Totally possible Im just doing something wrong
a
ya it yells at you if you try to define
``List[Nothing]``
- you just use
``Nothing``
and it accepts an incoming list
d
woah
a
oh shoot - the issue that might be biting you is that if the output types in the list are not
``Nothing``
d
yeah
almost all of our current solids have
``Nothing``
input and output
a
ok ya that seems like an oversight
d
just want to make sure Im following - you're saying
``input_def=InputDefinition(Nothing)``
should work, but
``input_defs=[InputDefinition(Nothing)]``
does not work?
as far as being able to handle n number of
``Nothing``
inputs
a
input_def=InputDefinition(Nothing) should work, but input_defs=[InputDefinition(Nothing)] does not work?
No I am saying
``InputDefinition(List[Nothing])``
is a problem
you should be able to define as many separate
``Nothing``
input defs as you want - though likely you only really need a single one that you can fan-in to if you need
d
yes ^ exactly
this fan-in process - how is that handled within Dagster?
a
for real types we just put the data in a list
d
ah okay - and what you're saying is because in this case it's a
``Nothing``
type, that
``List[Nothing]``
fan-in process doesn't work as expected
is that right
a
So I’m guessing (havent repro-ed) based on the fact that you said you tried it and it didnt work that:
Copy code
``````x = solid_a() # returns Nothing
y = solid_b() # returns Nothing
z = solid_c() # returns int (oh no!)

dbt_run(on_complete=[x, y, z]) # fails because thers an int in the list

dbt_run(on_complete=[x, y]) # works fine``````
if its easy for you to send over the error you are seeing that would be great
otherwise ill try to repro here in a little bit
d
yeah let me try something closer to what you posted. tbh I'm not familiar with this
``on_complete``
param
Ive just been modeling dependencies using the way specified in the docs
``solid_a(solid_b())``
a
oh thats just using keyword args
``dbt_run([x, y])``
is the same
👍 1
``on_complete``
just my imagined
``InputDefinition('on_complete', Nothing)``
d
oh wow you're right this totally works
👍 1
👍
I think the piece I was missing was the fanning in of solid dependencies
Is that documented somewhere?
m
poorly
oh, hah, maybe not
d
Lol