https://dagster.io/ logo
#ask-community
Title
# ask-community
s

sar

12/12/2022, 8:59 PM
Trying to follow this doc https://docs.dagster.io/guides/dagster/airbyte-ingestion-as-code (great writeup by the way) however, i’m not sure how to configure the values under
Streams
in the Airbyte UI : • Destination Namespace • Namespace Custom Format • Destination Stream Prefix Tried looking through the docs/code but wasn’t very clear to me.
🤖 3
r

rex

12/12/2022, 9:52 PM
I suggest that you repost your question in #dagster-airbyte
b

ben

12/12/2022, 10:17 PM
Hi @sar, you can configure the destination namespace as follows:
Copy code
AirbyteConnection(..., destination_namespace=AirbyteDestinationNamespace.SAME_AS_SOURCE)
valid values are
AirbyteDestinationNamespace.SAME_AS_SOURCE
,
AirbyteDestinationNamespace.DESTINATION_DEFAULT
, or a custom string (the same format that you would use in the Airbyte UI)
Right now, we don’t have support for destination prefixes, but this PR (https://github.com/dagster-io/dagster/pull/11057) should enable you to set them as an optional argument for a connection. Hopefully this will be out with this week’s release.
I’m noticing now that the autogenerated
AirbyteConnection
API docs don’t show the constructor’s documentation - will push a fix for that, but here is a link to those docs: https://dagster-fqt9qd90e-elementl.vercel.app/_apidocs/libraries/dagster-airbyte#dagster_airbyte.AirbyteConnection
s

sar

12/13/2022, 2:03 PM
Thanks Ben for all the info
@ben Am I correct in assuming when specifying an
AirbyteConnection
that a
stream_config
needs to be specified as well? The reason I ask because if you’re setting up a db sync, you’d have to get a list of tables in that db and specify each one of those streams which would be quite tedious if you want to sync it all. Additionally, if you end up adding tables, you’d have to go back to your code and add those streams as well. Am I missing something?
b

ben

12/16/2022, 3:25 PM
Yes, right now we require that users explicitly provide stream config. One option could be for us to discover all streams at apply-time, so that users could just say to sync all streams with a particular sync mode. It’s my understanding that Airbyte doesn’t support detecting/enabling new tables by default - I think the best we could do here would be to try to reread the schema and enable the new stream on apply
s

sar

12/19/2022, 2:36 PM
It would be great to discover all streams at apply-time and sync them with a particular sync mode to start with.