Ok so the above fan in attempt didn t work due to <https git dagster #announcements

Ok so the above fan in attempt didn't work due to ...

Danny

07/03/2020, 6:43 PM

Ok so the above fan in attempt didn't work due to https://github.com/dagster-io/dagster/issues/2679. Is there an alternative approach where I could fan in multiple solids in a pipeline, some of which may be skipped, and then continuing to execute other solids downstream of the fan in?

Copy code

A
   / \
  B   C
   \ /
    D
    |
    E

i.e. D defines the input from C as optional, and if C is skipped then D and E still run?

max

07/03/2020, 6:58 PM

I think you could use a single solid that accepted many inputs (some optional) and then yielded a list types result

Danny

07/03/2020, 7:01 PM

Tried that and documented why it failed in the issue I linked https://github.com/dagster-io/dagster/issues/2679. The

fan_in

solid accepted many optional inputs and just yielded the first one. Problem was that dagster enforces that as soon as a solid is skipped, anything that takes its outputs downstream is automatically skipped, even if those downstream solids mark those inputs as optional.

Danny

07/03/2020, 8:33 PM

@max as a workaround I'm planning to stop using conditional outputs in the pipeline. This way no steps get skipped, and they'll just need to have extra code to detect if they have no work to do and should return immediately. The implementation for this is pretty involved so it would be great to have an official opinion about other alternatives. 🙏

max

07/03/2020, 11:25 PM

yep, that makes sense. the downstream behavior certainly feels like a bug

👍 1

2 Views

Open in Slack

Previous Next