This channel is a place where you can get the latest updates on using Dagster with Airbyte, ask questions, and discuss with other members of the community. Looking to get started using Dagster with Airbyte? • ✨ Check out our getting started with Airbyte guide • 📚 Take a look at our API docs for `dagster-airbyte`

@Dusty Shapiro has joined the channel

@Adam Bloom has joined the channel

@ben has joined the channel

@sandy has joined the channel

@yuhan has joined the channel

@Blake Hughes has joined the channel

@Dagster Jarred has joined the channel

@daniel has joined the channel

@Emil Sundman has joined the channel

@Alexis Manuel has joined the channel

We are just getting started into our Airbyte/Dagster journey, so I’m sure you’ll be hearing from me in this channel 🙃

@geoHeil has joined the channel

@Arthur has joined the channel

@Matias Ferrero has joined the channel

@Dario Forti has joined the channel

@Mora forti has joined the channel

@Juan has joined the channel

@Cameron Gallivan has joined the channel

@pdpark has joined the channel

@Jason has joined the channel

I am curious, when it comes to testing Dagster and Airbyte locally, how people have that setup via Docker? Would the local AIRBYTE_HOST be

localhost

or the docker container name? and port? I found the answer:

<http://host.docker.internal:8000>

If they're both running via docker, they'd need to be in the same docker bridge network to use container names. Otherwise, localhost + a bound host port would work. I haven't had a use case to run airbyte locally though - we have a dev and prod deployment of airbyte (and dagster) in two different k8s clusters. I'll usually just commandeer the dev deployment for local testing (stopping schedules/sensors in that environment's dagster deployment as needed to avoid conflicts)

@Simon Späti belongs in here

@Simon Späti has joined the channel

Hi all - wanted to highlight a couple things on the horizon for

dagster-airbyte

First - a big thank you to @Adam Bloom who contributed filtering to drastically cut down on load times when generating Airbyte assets. In addition to the ability to load Dagster assets from your Airbyte instance or Octavia CLI files, we are experimenting with the ability to manage Airbyte Connections from Python code, which we plan on rolling out later this week. We’ll post more details on how you can try it out along with the release. We’re eager for feedback on how we can make using Dagster with Airbyte even better - if there’s improvements you’d like to see to the APIs, the metadata we show in Dagit, documentation, etc, let us know!

is it worth pausing on all airbyte-dagster development in our workflows till the new release comes out because it looks mega

Question related to my recently setup Airbyte integration in Dagster: We’re currently loading our assets from our airbyte instance and all seemed well, until an upstream table was removed from our replication. Now, the sync runs fine, but that specific table/asset throws a

DagsterStepOutputNotFoundError

during the materialization step post sync. Is there something we are missing when it comes to tables that were dropped in our source DB?

Is there anyway to set downstream dependancies of the assets returned in

load_assets_from_airbyte_instance

?

new to both airbyte and dagster, but having trouble with even the getting started simple example:

assets = with_resources(
  load_assets_from_airbyte_project(project_dir=/my/ab/proj),
  {
    "airbyte": airbyte_resource.configured({ "host": "localhost", "port": "8000", })
  },
)

looks like the load assets returns a

CacheableAssetsDefinition

which isn't iterable