Jordan
08/22/2022, 3:26 PMbuild_assets_job
which allowed me to define the resources allocated to the job at the time of its definition. So it was much easier to define the resources without having to give them different names to differentiate them. Since version 1.0, I use define_asset_job
,so the resources are defined in the repository and not in the job directly, it is necessary to call them differently to avoid the following error:
dagster._core.errors.DagsterInvalidDefinitionError: Conflicting versions of resource with key 'cred' were provided to different assets. When constructing a job, all resource definitions provided to assets must match by reference equality for a given key.
The solution I use is to create a function that creates a resource for me and I use the getattr
function in the asset to retrieve the resource. Since I have a lot of resources the code becomes unreadable with this solution. Is there a better solution that allows me to create resources with the same name and just use context.resources.resource
. Thanks in advance
def build_asset(name : str) -> AssetsDefinition:
@asset(name = f"asset_{name}", required_resource_keys={f"resource_{name}"})
def asset_def(context):
return getattr(context.resources,f"resource_{name}")
return asset_def
def build_job(name : str, assets : AssetsDefinition):
return define_asset_job(
name=f"job_{name}",
selection=AssetSelection.assets(*assets)
)
def build_resource(name : str)-> ResourceDefinition:
@resource()
def resource_def(_):
return f"foo_{name}"
return {f"resource_{name}" : resource_def}
@repository
def my_repo():
assets=[]
jobs=[]
for value in ['A','B','C']:
asset = with_resources([build_asset(value)], build_resource(value))
job = build_job(value, asset)
assets.append(asset)
jobs.append(job)
return [*assets, *jobs]
chris
08/22/2022, 8:50 PMwith_resources
pattern definitely makes it more annoying to specify different resources for the same key, but that's partially because it's expected to be kind of a niche use case. I'm wondering what your use case is for providing a bunch of different resources to the same key - perhaps getting that sorted out might be a faster solution