Hi everyone. I am trying to write an op with an optional input of type dagster_pyspark.types.DataFrame. I implemented this entrance in the ins dictionary as

'optional_input': In(DataFrame, default_value=None)

. This works when I pass a DataFrame to the op. However, if I pass no input, it raise this error:

DagsterTypeCheckDidNotPass: Type check failed for step input "optional_input" - expected type "PySparkDataFrame". Description: Value of type <class 'NoneType'> failed type check for Dagster type PySparkDataFrame, expected value to be of Python type DataFrame.

Besides, if I implement it as

'optional_input': In(Any, default_value=None)

, it would raise an error when passing a DataFrame to the op. Can someone help me? Thanks!

Hi @Bernardo Cortez -- what error were you seeing for the

In(Any, ...)

solution? It's surprising to me that this would cause a problem. In any case, you should be able to do:

from typing import Optional

@op(ins={"optional_input": In(Optional[DataFrame], default_value=None)})
def my_op(optional_input):
   ...

The 

In(Any, ...)

  solution raise the following error:

CheckError: Failure condition: Inputs of type <dagster.core.types.dagster_type._Any object at 0x7f18b6472e80> not supported. Please specify a valid type for this input either in the solid signature or on the corresponding InputDefinition.

ah interesting -- you can also leave the Dagster type blank for that sort of case (which will default it to not doing type checking)

Well, if I do not specify a data type, I get the following error:

CheckError: Failure condition: Inputs of type <dagster.core.types.dagster_type._Any object at 0x7efe99ec9e80> not supported. Please specify a valid type for this input either in the solid signature or on the corresponding InputDefinition.

Oh I believe that's an issue that's coming from the IOManager code. It looks like you're using the Parquet IOManager from the hacker news example, which only accepts two specific input types on the input definition (pyspark dataframe and string). You'll need to modify the example code to get this setup working.

Yes! Thank you. This was the root of the problem!