https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Chris Nogradi

05/26/2023, 6:54 PM
If I have an asset that is downstream of another that uses an iomanager which is based on the UPathIOManager which is capable of returning either a single obj or a dict of objs, how do I specify in the asset's arguments the type of the expected input since it would be a union of the dataframe and a dict of dataframes? If I use this: @asset() def myasset(context, upstream_asset: Union[pa.dataframe.Dataset, Dict[str, pa.dataframe.Dataset]): return upstream_asset I get an error that the Union is not a DagsterType or Python type. However without the Union (pa.dataframe.Dataset) it works fine until the iomanager returns a dict of objects. ok I see that in https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_utils/typing_api.py#L119 Unions are not supported. However docs say that PEP484 should be supported and it looks like Unions are in there. Thoughts? @sean
The error is this: File "\git\cbm+\d4- input_defs.append(input_def.combine_with_inferred(inferred_props[input_def.name])) File "\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_core\definitions\input.py", line 300, in combine_with_inferred dagster_type = _checked_inferred_type(inferred) File "\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_core\definitions\input.py", line 348, in _checked_inferred_type raise DagsterInvalidDefinitionError( The above exception was caused by the following exception: dagster._core.errors.DagsterInvalidDefinitionError: Invalid type: dagster_type must be an instance of DagsterType or a Python type: got typing.Union[pyarrow._dataset.Dataset, typing.Dict[str, pyarrow._dataset.Dataset]]. Stack Trace: File "\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_core\definitions\input.py", line 345, in _checked_inferred_type resolved_type = resolve_dagster_type(inferred.annotation) File "\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_core\types\dagster_type.py", line 910, in resolve_dagster_type raise DagsterInvalidDefinitionError( orchestrator\.venv\lib\site-packages\dagster\_core\definitions\decorators\op_decorator.py", line 438, in resolve_checked_op_fn_inputs Which, to me, is strange since load_input() for upath_io_manager.py returns Union[Any, Dict[str, Any]] which is incompatible with the code in typing_api.py. @sean
https://github.com/dagster-io/dagster/issues/5115 Looks like this is already marked as an issue but has not been addressed for a while. Any reason for not addressing and is there a simple work around using DagsterType? I have tried various attempts of mapping typing.Union to a dagster_type but can't figure it out. Tagging @claire since she picked the issue up a long time ago.