Mike Jones
08/10/2022, 6:28 PMDagsterInvalidDefinitionError
when I run code samples from the documentation. Details in thread.dagster._core.errors.DagsterInvalidDefinitionError: Asset key AssetKey(['downstream_asset']) is defined multiple times. Definitions found in modules: test.
File "C:\Users\mike.jones\Projects\dagster\venv\lib\site-packages\dagster\_grpc\server.py", line 227, in __init__
self._loaded_repositories = LoadedRepositories(
File "C:\Users\mike.jones\Projects\dagster\venv\lib\site-packages\dagster\_grpc\server.py", line 101, in __init__
loadable_targets = get_loadable_targets(
File "C:\Users\mike.jones\Projects\dagster\venv\lib\site-packages\dagster\_grpc\utils.py", line 33, in get_loadable_targets
else loadable_targets_from_python_file(python_file, working_directory)
File "C:\Users\mike.jones\Projects\dagster\venv\lib\site-packages\dagster\_core\workspace\autodiscovery.py", line 27, in loadable_targets_from_python_file
return loadable_targets_from_loaded_module(loaded_module)
File "C:\Users\mike.jones\Projects\dagster\venv\lib\site-packages\dagster\_core\workspace\autodiscovery.py", line 113, in loadable_targets_from_loaded_module
asset_group_from_module_assets = AssetGroup.from_modules([module])
File "C:\Users\mike.jones\Projects\dagster\venv\lib\site-packages\dagster\_core\definitions\asset_group.py", line 336, in from_modules
assets, source_assets = assets_and_source_assets_from_modules(modules, extra_source_assets)
File "C:\Users\mike.jones\Projects\dagster\venv\lib\site-packages\dagster\_core\definitions\load_assets_from_modules.py", line 61, in assets_and_source_assets_from_modules
raise DagsterInvalidDefinitionError(
dagit==1.0.2
dagster==1.0.2
dagster-aws==0.16.2
dagster-gcp==0.16.2
dagster-graphql==1.0.2
dagster-pandas==0.16.2
Aaron
08/10/2022, 6:34 PMMike Jones
08/10/2022, 6:36 PMAaron
08/10/2022, 6:45 PMclaire
08/10/2022, 6:46 PMwith_resources
--this function will actually return your assets, but bound to the resources provided. This is the cause of the error, because the file now contains both the original assets and the assets bound to resources. One thing you can do here is put your with_resources
call within a repository:
@repository
def my_repository():
return [
with_resources(
[upstream_asset, downstream_asset],
resource_defs={"io_manager": s3_pickle_io_manager, "s3": s3_resource},
)
]
Or just bind the resources directly to your asset:
@asset(resource_defs={...})
def my_asset():
...
Dagster Bot
08/10/2022, 6:48 PM@dagster_bot <issue|docs|discussion> <title>
claire
08/10/2022, 6:48 PMDagster Bot
08/10/2022, 6:48 PMMike Jones
08/10/2022, 6:52 PMFrank Lu
09/15/2022, 10:09 PMclaire
09/15/2022, 10:13 PMFrank Lu
09/16/2022, 7:23 PMclaire
09/16/2022, 9:16 PMsandy
09/16/2022, 9:26 PMwith_resources
in a module at the top level of your asset folder and then importing the assets with resources into your repository?Frank Lu
09/16/2022, 9:26 PM