fahad
07/07/2023, 8:44 PM@asset
def output_1():
return 1
@op
def add_1(raw_num: int):
print(raw_num + 1)
@graph
def adder_graph():
add_1(output_1.to_source_asset())
resources = {"io_manager": fs_io_manager}
test = Definitions(
assets=load_assets_from_current_module(group_name="Adder", key_prefix=["reference", "adders"]),
resources=resources,
jobs=[adder_graph.to_job(resource_defs=resources,)]
)
The issue appears to be the invocation via load_assets_from_current_module
which adds a key prefix to the asset, and therefore changes the output location of the asset. When I try to consume the asset via <http://output_1.to|output_1.to>_source_asset()
it tries to look for the asset data at the wrong path:
dagster._core.errors.DagsterExecutionLoadInputError: Error occurred while loading input "raw_num" of step "add_1":
The above exception was caused by the following exception:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/fahad/.dagster/storage/output_1'
Looking at the actual storage path for output_1
it is being materialized to /Users/fahad/.dagster/storage/reference/adders/output_1
. This seems to make sense because load_assets-from_current_module
is creating a copy of the definition. That being said, is there a way to tell to_source_asset
that I want to apply an asset key prefix?jamie
07/07/2023, 9:23 PM@asset(
key_prefix=["reference"]
)
def output_1():
return 1
this is probably worth opening a github issue - can you open one and include the details of your use case? https://github.com/dagster-io/dagster/issues/new/choosefahad
07/07/2023, 10:08 PM