https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jeremy

06/06/2023, 3:59 PM
Hi all, I'm wanting to reuse an op to load multiple datasets from different datastores. My code looks something like
Copy code
@op(required_resources={"file_system"})
def read_files(context):
    ...

first_dataset = AssetsDefinition.from_op(read_files)

second_dataset = AssetsDefinition.from_op(read_files)
where I want to be able to define the
"file_system"
resource for each of the two assets. I've tried
config_mapping
with and without
jobs
,
with_resource
but it seems only the resource defined in
Definition
is used. Maybe my implementaiton is wrong or maybe I've got entirely the wrong approach. Any suggestions would be appreciated.
s

sandy

06/06/2023, 4:31 PM
hi Jeremy - have you considered an "asset factory"?
Copy code
def make_dataset_asset(name, file_system_resource_key):
    @asset(name=name, required_resource_keys={file_system_resource_key})
    def _asset(context):
        ...

    return _asset


first_dataset = make_dataset_asset("asset1", "fs1")
second_dataset = make_dataset_asset("asset2", "fs2")
j

Jeremy

06/06/2023, 5:06 PM
Thanks for the reply sandy. I think I had copy and pasted an asset previously when I was short on time, and a factory would be a little more elegant. But, it does mean rewriting any assets I want to reuse or knowing I want to reuse them a priori, which seems like a big blow to re-usability.
2 Views