https://dagster.io/ logo
Join Slack
Powered by
Trying to follow this doc <https://docs.dagster.io...
# ask-community
s
Trying to follow this doc https://docs.dagster.io/guides/dagster/airbyte-ingestion-as-code (great writeup by the way) however, i’m not sure how to configure the values under 
Streams
in the Airbyte UI : • Destination Namespace • Namespace Custom Format • Destination Stream Prefix Tried looking through the docs/code but wasn’t very clear to me.
🤖 3
r
I suggest that you repost your question in #dagster-airbyte
b
Hi @sar, you can configure the destination namespace as follows:
Copy code
AirbyteConnection(..., destination_namespace=AirbyteDestinationNamespace.SAME_AS_SOURCE)
valid values are 
AirbyteDestinationNamespace.SAME_AS_SOURCE
, 
AirbyteDestinationNamespace.DESTINATION_DEFAULT
, or a custom string (the same format that you would use in the Airbyte UI)
Right now, we don’t have support for destination prefixes, but this PR (https://github.com/dagster-io/dagster/pull/11057) should enable you to set them as an optional argument for a connection. Hopefully this will be out with this week’s release.
I’m noticing now that the autogenerated 
AirbyteConnection
API docs don’t show the constructor’s documentation - will push a fix for that, but here is a link to those docs: https://dagster-fqt9qd90e-elementl.vercel.app/_apidocs/libraries/dagster-airbyte#dagster_airbyte.AirbyteConnection
s
Thanks Ben for all the info
@ben Am I correct in assuming when specifying an 
AirbyteConnection
that a 
stream_config
needs to be specified as well? The reason I ask because if you’re setting up a db sync, you’d have to get a list of tables in that db and specify each one of those streams which would be quite tedious if you want to sync it all. Additionally, if you end up adding tables, you’d have to go back to your code and add those streams as well. Am I missing something?
b
Yes, right now we require that users explicitly provide stream config. One option could be for us to discover all streams at apply-time, so that users could just say to sync all streams with a particular sync mode. It’s my understanding that Airbyte doesn’t support detecting/enabling new tables by default - I think the best we could do here would be to try to reread the schema and enable the new stream on apply
s
It would be great to discover all streams at apply-time and sync them with a particular sync mode to start with.
Open in Slack
PreviousNext
Powered by