Hey all Is it possible to provide optional inputs to a graph

Question

Hey all Is it possible to provide optional inputs to a graph Basically looking for something to hook into a graph of graphs that has some conditional OPs ``` graph description= Optional sub graph tags= dagster k8s config k8s tags def merge variants graph optional input 1 Optional Any optional input 2 Optional Any gt Any merge op optional input 1 optional input 2 ``` Doing this throws the following error ```Operation name PipelineExplorerRootQuery Message Optional Any Path pipelineSnapshotOrError solidHandles 0 solid definition inputDefinitions 0 type Locations line 238 column 5 ```

fahad · Answer

Update The actual input can be `None` but dagster cannot handle the `Optional` type annotation on a graph input

prha · Answer

Can you include the definition of `merge op` The following works for me ``` op def show number context number Optional Int if number is None No number else f Number = number graph def my graph optional input 1 Optional Int optional input 2 Optional Int show number alias one optional input 1 show number alias two optional input 2 ```

fahad · Answer

For sure let me minimize the example a bit more real quick

fahad · Answer

This is somewhat what I have ``` op description= Optional out= optional output Out Any is required=False def yield optional optional input Any gt Any if optional input is not None yield Output optional input optional output op description= Merges def merge op context OpExecutionContext vcfs list Path for vcf in vcfs <http context log info|context log info> vcf graph description= Merges multiple variant calling pipeline VCFs into a single MAF file output tags= dagster k8s config k8s tags def merge variants graph vcf1 Optional Path vcf2 Optional Path merge op yield optional vcf1 yield optional vcf2 ```

fahad · Answer

``` vcf1 Optional Path vcf2 Optional Path ``` Removing that type annotation allows dagit to pick up the graph Otherwise I get the graphql error

prha · Answer

I was able to load the following swapped `Int` instead of `Path` to simplify ``` op out= optional output Out Any is required=False def yield optional optional input Any gt Any if optional input is not None yield Output optional input optional output op description= Merges def merge op context vcfs List Int for vcf in vcfs <http context log info|context log info> vcf graph def merge variants graph vcf1 Optional Int vcf2 Optional Int merge op yield optional vcf1 yield optional vcf2 ```

fahad · Answer

Let me try `Int`

fahad · Answer

Weird still getting ```Operation name PipelineExplorerRootQuery Message <http Optional Int|Optional Int> Path pipelineSnapshotOrError solidHandles 0 solid definition inputDefinitions 0 type Locations line 239 column 5 ```

fahad · Answer

I think this is more related to how I m calling the graph with optional inputs then Might be able to work with that thanks

prha · Answer

Sure let me know if there s an example I can help dig into

fahad · Answer

Thanks I think i m running into some confusion because ops graphs can conditionally output optional data but cannot take these conditional outputs as python optional inputs A dagster optional Out doesn t truly map to a python Optional type

fahad · Answer

Instead the graph op just doesn t run at all

prha · Answer

right the optional inputs vcf1 vcf2 mean the Python type where it s a value that is an `int` or `None` Those get mapped to the actual inputs in `yield optional` which conditional yield an Output

prha · Answer

Consider the following graph ``` graph def example graph vcf1 Optional Int vcf2 Optional Int one = yield optional alias a vcf1 two = yield optional alias b vcf2 process one one process two two ``` where `process one` `process two` are also ops If `vcf1=3` and `vcf2=None` then the aliased op a and the aliased op b will both execute Aliased op a will emit an output and aliased op b will not Then `process one` will execute and `process two` will skip

fahad · Answer

Makes sense which means I should probably use the fan in paradigm rather than optional inputs if I know the upstream graph op is production output conditionally

fahad · Answer

Unless there s some other way to force execution with python `None` when upstream produces no output

prha · Answer

So Optional here refers to the value that s being passed in but non required Outs refer to the conditional branching <https docs dagster io concepts ops jobs graphs jobs graphs conditional branching>

prha · Answer

I think if you wanted to force the downstream execution you would `yield Output None ` in the `else` case ``` op def yield optional optional input Any gt Any if optional input is not None yield Output optional input optional output else still yield an output it will just be None yield Output optional input optional output ```

fahad · Answer

Oh good call