Prratek Ramchandani03/14/2021, 12:50 AM
fetches some data from an API, and uses an IO Manager that persists the data to a JSON file in GCS. The path at which to store the file is specified as config for IO Manager as described in the docs here.
then loads that file from GCS to BigQuery, but I’m not sure how to pass the GCS URI between the two solids. I can’t have
use the same IO Manager and have the IO Manager’s
handle locating the file because I don’t want to “load” it to perform computation.
Is there a way for
to access that IO manager config to return it as on output? Also, is there a better way I could model this series of tasks?
owen03/15/2021, 4:45 PM
function to simply return the URI (
), instead of actually loading the data from GCS. So the protocol for this IOManager would be: on output, store data into the location defined by the config, and on input, simply retrieve the path that this data was stored in.
context.upstream_output.config["<whatever you call the uri>"]
Prratek Ramchandani03/15/2021, 4:54 PM