https://dagster.io/ logo
#announcements
Title
# announcements
a

Andy H

04/14/2021, 11:25 PM
I have a question about input dependencies when using default values and optional outputs. In the following example, my
do_computation
solid will not be called even though I’ve marked the output and inputs as optional. I believe I’m just misunderstanding how this is meant to work, and could use a bit of help understanding.
s

schrockn

04/14/2021, 11:37 PM
Oh this is interesting. To be clear in this cause you would expect
do_computation
to be called even through its upstream computation failed, and you would expect the default to be applied?
a

Andy H

04/14/2021, 11:40 PM
@schrockn Yes, that was my assumption, but I can also understand why you would want downstream dependency to fail on failed upstream dependency. I would prefer to use the default if provided, but it's reasonable that it would work either way.
👍 1
s

schrockn

04/14/2021, 11:41 PM
Gotcha. Our mental model of defaults is applying them at the start of execution. Very specifically when we compile our execution plan.
a

Andy H

04/14/2021, 11:42 PM
Yes it seems like it would get tricky to represent a stable graph model if you can just yoink a chunk out of it post-execution
Thanks for the explanation. The way I was explaining it to my team was that in this case,
do_computation
could be called as
do_computation()
and would supply it's own default, but this would need to be constructed from the pipeline entrypoint.
That's what my testing has informed me of, anyhow haha
s

schrockn

04/14/2021, 11:47 PM
I think we need to do better to support this type of pattern anyways so I’m interested in your concrete use case
a

Andy H

04/14/2021, 11:57 PM
Essentially, our pipeline is designed so that each solid is adding annotation to a larger dataset. We don't mind if one of the solids crashes out, so we want to add empty annotations for that leg of the pipeline in our final solid which does the aggregation of all new annotations. I'm not sure if that makes sense.
s

schrockn

04/15/2021, 12:00 AM
yup make sense
*makes
@sandy ^--- interesting use case to catalog
👍 1
a

Andy H

04/15/2021, 12:03 AM
Most of the scientific pipelines we work on, the need to collect any data from the pipeline is greater than guaranteeing a successful pipeline run, in general, if that helps clarify the case.
We could also just write better code and return defaults from our solids when an exception raises 🙂