https://dagster.io/ logo
#integration-airbyte
Title
# integration-airbyte
g

Gabriel Samson

03/03/2023, 8:32 AM
👋 Building out a PoC for our team to adopt Dagster. It's amazing so far. Is it expected / by design for Airbyte assets to have an output value of
None
when defined with
load_assets_from_airbyte_instance
? I would expect the output to be the sync job output like the output value of
airbyte_sync_op
. tl;dr I am trying to auto-define an asset that represents a sync to S3. Then that S3 sync feeds into an asset which gets the S3 object based on the Airbyte sync results:
Copy code
airbyte sync to s3 -> fetch from s3 -> processing in df
Or is the only way to achieve something like this with a job that first calls the Airbyte sync op, then materializes an asset from the fetch S3 object. As described here
s

sandy

03/03/2023, 4:55 PM
@Gabriel Samson if you provide an
io_manager_key
, that IO manager will be used to load the values for downstream assets, even though the airbyte sync op returns
None
cc @ben
g

Gabriel Samson

03/03/2023, 7:16 PM
@sandy I believe this might have only worked with a database table as opposed to an S3 object. I can re-confirm later today though. Thanks!
3 Views