Gabriel Samson
03/03/2023, 8:32 AMNone
when defined with load_assets_from_airbyte_instance
? I would expect the output to be the sync job output like the output value of airbyte_sync_op
.
tl;dr I am trying to auto-define an asset that represents a sync to S3. Then that S3 sync feeds into an asset which gets the S3 object based on the Airbyte sync results:
airbyte sync to s3 -> fetch from s3 -> processing in df
Or is the only way to achieve something like this with a job that first calls the Airbyte sync op, then materializes an asset from the fetch S3 object. As described heresandy
03/03/2023, 4:55 PMio_manager_key
, that IO manager will be used to load the values for downstream assets, even though the airbyte sync op returns None
cc @benGabriel Samson
03/03/2023, 7:16 PM