Hey guys, we have a use case where we often need to transfer files between assets. For example we generate a GeoJSONSeq file in one asset then one asset will generate tiles and another will upload that to bigquery. How can we transfer temporary files using the IOManager?
02/02/2023, 8:58 PM
Hi @Mitchell Fasanya - you can specify different io managers for the inputs and outputs of your asset. For example, you could use one io manager to store the GeoJSONSeq file and load it into the tiling asset. Then you can return the tiles from the asset and set a bigquery io manager as the io manager responsible for that output.
it’d be like this example https://docs.dagster.io/concepts/io-management/io-managers#per-asset-io-manager
here the s3_io_manager is used to store the output of upstream_asset and load it as the input to downstream_asset. then the fs_io_manager is used to store the output of downstream_asset