I created a general purpose fan-in solid:

# Adapted from:
# <https://github.com/dagster-io/dagster/blob/e59a93a5d5e7e15a19086a7e773e2357f174e9b1/examples/legacy_examples/dagster_examples/toys/fan_in_fan_out.py>
#
@solid(
    input_defs=[
        InputDefinition('main_fan', Any),
        InputDefinition('other_fans', Optional[List[Any]], default_value=None),
    ],
    output_defs=[
        OutputDefinition(name='main_fan_return_value', dagster_type=Any)
    ],
)
def fan_in(_, main_fan, other_fans):
    yield Output(main_fan, 'main_fan_return_value')

To be used like this:

@pipeline
def my_pipeline():
    main_id, other_id = solid2(solid1()) # other_id is an optional output
    fan1 = solid3(main_id)
    fan2 = solid4(other_id)
    main_id = fan_in(main_id, [fan1, fan2])
    solid5(main_id)

This almost works. Only problem is that if

other_id

is not outputted by solid2,

fan_in

doesn't run because Dagster complains its dependencies (fan2 from solid4) didn't produce an output. Clearly I'm using Optional incorrectly here, but can't really see how to fix this. Any advice?

maybe Optional[List[Optional[Any]]?

Neither one worked blob frown

hmmmmm, unsure then

Looks like that's only for OutputDefinition, not InputDefinition https://docs.dagster.io/_apidocs/solids#dagster.InputDefinition

That takes a default value, though -- I think the issue is that it's not quite a default?

You mean the problem is that it's being set to

None

? I'll try changing, one sec

No -- the list overall is present, it's just missing a value

Not sure I follow. The

other_fans

list in this my_pipeline example would always be present as you say, since even if solid4/fan2 doesn't run, the list will still contain fan1. So I don't really see

default_value

coming into play at all? What appears to be happening is that either I'm using Optional wrong (tried all the combinations you suggested, still not working), or Optional can't denote that the items in a list input are optional, only that the input itself is optional. But once the input is present, like the list is in this case, Dagster seems to enforce that any items in that list are actually produced. If the upstream solid that was supposed to produce them didn't run, the fan_in solid doesn't get run.

My suspicion is it's the latter

Got it

It may, unsure

(you can't currently denote the optionality of elements in a list trivially)

Do you mean its hard to do because this hasn't yet been implemented in the current InputDefinition API (but might be easy to implement in Dagster), or because implementing this into the API would be hard to do?

Unsure

👍

weird, even though they're recognized as optional the solid breaks if they're missing?

Yes. This confirms our suspicion above: Dagster enforces that if any of those optional inputs come from solids that were skipped, the fan_in solid gets skipped as well.

Unsure

Ok, I'll create it then