Cary Murray
04/03/2023, 8:26 PMAssetsDefinition.from_op
in an asset factory. In order to not have number-indexed keys in my config for each of the assets made from my factory, I am calling an op factory within my asset factory (to rename the op). Is there a way to do this renaming at the asset factory level?sandy
04/04/2023, 12:15 AMAssetsDefinition.from_op
?
Also, do you need to use`AssetsDefinition.from_op` at all if you're already using an asset factory?Cary Murray
04/04/2023, 12:52 PMdef transform_asset_factory(table_name: str = None) -> AssetsDefinition:
transformed = transform_op_factory(
name=f"transform_{table_name}",
)
return AssetsDefinition.from_op(
transformed,
keys_by_input_name={f"table": AssetKey(f"assigned_{table_name}")},
keys_by_output_name={f"result": AssetKey(f"transformed_{table_name}")},
)
If there was a way to use the asset factory’s table_name
parameter to name the asset, I would prefer to go that route instead of having to define a new op that has the desired name that contains the table_name
.sandy
04/04/2023, 3:29 PMdef transform_asset_factory(table_name: str = None) -> AssetsDefinition:
@asset(
name=f"transformed_{table_name}",
ins={"assigned_table": AssetIn(key=f"assigned_{table_name}")}
)
def _asset(assigned_table):
# whatever logic is inside transform_op_factory
return _asset
Cary Murray
04/14/2023, 2:35 PMtransformed
as an op for backwards compatability (we are currently moving our pipelines over from ops+graphs to assets). Is there a way to “duplicate” and rename an op that backs an asset?sandy
04/14/2023, 6:06 PMAssetsDefinition.from_node
. If you want to file an issue for this, we might be able to get to it at some point.
another idea here that could help is that you can invoke your op like a regular function, inside your asset. e.g.
def transform_asset_factory(table_name: str = None) -> AssetsDefinition:
@asset(
name=f"transformed_{table_name}",
ins={"assigned_table": AssetIn(key=f"assigned_{table_name}")}
)
def _asset(context, assigned_table):
transform_op.compute_fn(context, assigned_table)
return _asset