Eric Larson
06/24/2022, 4:58 PMExperimental#
If two assets are in the same group and the upstream asset has a multi-segment asset key, the downstream asset doesn't need to specify the full asset key when declaring its dependency on the upstream asset - just the last segment.
I am specifying the key_prefix
for my multi segment asset key in the @asset(ins=)
and it worked in 0.15.0 but 0.15.1 is not respecting it, and picking up other assets with the same base asset_key
sandy
06/24/2022, 5:30 PMEric Larson
06/24/2022, 5:49 PM@asset(
compute_kind='SFTP',
key_prefix='l2_commercial',
group_name='l2',
)
def sftp_files():
return ['files']
@asset(
partitions_def=state_parts,
key_prefix='l2_commercial',
ins={
"sftp_files": AssetIn(key=sftp_files.asset_key),
},
group_name='l2',
)
def sftp_filename(context, sftp_files: List[str]) -> str:
return 'filename'
# In another Python module
@asset(
compute_kind='SFTP',
key_prefix='VM2Uniform',
group_name='l2',
)
def sftp_files(context) -> List[str]:
return ['files']
# in another module
sftp_assets = with_resources([
vm2uniform.sftp_files,
],
resource_defs = l2_resources,
)
vm2_assets = with_resources([
vm2uniform.sftp_filename,
],
resource_defs = l2_resources,
)
l2_com_assets = with_resources(
definitions=[
l2_commercial.sftp_filename,
],
resource_defs = l2_resources,
)
l2_commercial_job = define_asset_job(
name='Pipeline_Commercial',
selection=AssetSelection.assets(*l2_com_assets),
partitions_def=state_parts,
tags = {
"ecs/cpu": "512",
"ecs/memory": "2048",
},
)
@repository
def helm_repository():
return [
l2_commercial_job,
] + sftp_assets + vm2_assets + l2_com_assets
asset_in
then it uses that, otherwise it would use this new experimental convience that is in 0.15.1sandy
06/24/2022, 6:21 PMEric Larson
06/24/2022, 7:17 PMsandy
06/24/2022, 10:27 PM