19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
If I have an asset that is downstream of another t...
# ask-communityc
Chris Nogradi
05/26/2023, 6:54 PMIf I have an asset that is downstream of another that uses an iomanager which is based on the UPathIOManager which is capable of returning either a single obj or a dict of objs, how do I specify in the asset's arguments the type of the expected input since it would be a union of the dataframe and a dict of dataframes? If I use this:
@asset()
def myasset(context, upstream_asset: Union[pa.dataframe.Dataset, Dict[str, pa.dataframe.Dataset]):
return upstream_asset
I get an error that the Union is not a DagsterType or Python type. However without the Union (pa.dataframe.Dataset) it works fine until the iomanager returns a dict of objects.
ok I see that in https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_utils/typing_api.py#L119 Unions are not supported. However docs say that PEP484 should be supported and it looks like Unions are in there. Thoughts? @sean
Chris Nogradi
05/30/2023, 1:07 PMThe error is this:
File "\git\cbm+\d4- input_defs.append(input_def.combine_with_inferred(inferred_props[input_def.name]))
File "\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_core\definitions\input.py", line 300, in combine_with_inferred
dagster_type = _checked_inferred_type(inferred)
File "\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_core\definitions\input.py", line 348, in _checked_inferred_type
raise DagsterInvalidDefinitionError(
The above exception was caused by the following exception:
dagster._core.errors.DagsterInvalidDefinitionError: Invalid type: dagster_type must be an instance of DagsterType or a Python type: got typing.Union[pyarrow._dataset.Dataset, typing.Dict[str, pyarrow._dataset.Dataset]].
Stack Trace:
File "\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_core\definitions\input.py", line 345, in _checked_inferred_type
resolved_type = resolve_dagster_type(inferred.annotation)
File "\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_core\types\dagster_type.py", line 910, in resolve_dagster_type
raise DagsterInvalidDefinitionError(
orchestrator\.venv\lib\site-packages\dagster\_core\definitions\decorators\op_decorator.py", line 438, in resolve_checked_op_fn_inputs
Which, to me, is strange since load_input() for upath_io_manager.py returns Union[Any, Dict[str, Any]] which is incompatible with the code in typing_api.py. @sean
Chris Nogradi
05/30/2023, 2:30 PMhttps://github.com/dagster-io/dagster/issues/5115
Looks like this is already marked as an issue but has not been addressed for a while. Any reason for not addressing and is there a simple work around using DagsterType? I have tried various attempts of mapping typing.Union to a dagster_type but can't figure it out. Tagging @claire since she picked the issue up a long time ago.