Alessandro Facchin
07/07/2022, 10:08 AMAssetsDefinition.from_graph(my_graph)
, but I cannot find how to set an io_manager
on these assets different from the default one. Is this doable? Shall I edit the graph or can I act directly on the newly generated asset?
Thank you 🙏Jordan
07/07/2022, 10:22 AMwith_resources
function for each asset by passing the io_manager
you want in the resource_defs
argument.Alessandro Facchin
07/07/2022, 10:46 AMio_manager
for all other assets in the pipeline. I would ideally like to just change the io manager for these specific assets. For example, my situation resembles the following:
@repository
def my_repo():
my_new_asset = AssetsDefinition.from_graph(my_graph)
assets_with_standard_io = [asset1, asset2]
assets_with_parquet_io = with_resources(
[asset3, asset4],
resource_defs={"s3_parquet_io_manager": s3_parquet})
return [my_new_asset, assets_with_standard_io, assets_with_parquet_io]
Right now my_new_asset
is using the default io, which is io_manager
. I would like to switch and use the s3_parquet_io_manager
instead, without impacting asset1
and asset2
output_defs
in the graph and specify the io_manager_key
. I'll give this a tryDaniel Gafni
07/07/2022, 7:34 PMAlessandro Facchin
07/07/2022, 7:56 PM