Having some issues materializing Airbyte assets as described

Question

Having some issues materializing Airbyte assets as described by <https docs dagster io integrations airbyte loading airbyte asset definitions from an airbyte instance|this documentation> All indications is that it is able to connect to my AIRBYTE HOST but perhaps I m passing the wrong value Since Airbyte is deployed to the same cluster as my Dagster app instead of `localhost` as the host I m using `airbyte airbyte server svc` It definitely is able to connect but doesn t seem to be pulling assets

ben · Answer

Hi Dusty are you seeing any logs which look related at repo load time

Dusty Shapiro · Answer

nope shake head

Dusty Shapiro · Answer

We have DBT Assets which are apart of this module and those are populating

Dusty Shapiro · Answer

Actually I whittled down our example to just Airbyte and now I m seeing ``` Request to Airbyte API failed HTTPConnectionPool host= airbyte airbyte server svc port=8000 Max retries exceeded with url ```

Dusty Shapiro · Answer

I m curious what the host should be

Adam Bloom · Answer

< Dusty Shapiro> maybe try `airbyte airbyte server svc lt namespace gt svc cluster local` You mentioned same cluster but if airbyte and dagster are in different namespaces you ll need to include the full internal k8s hostname that includes the airbyte namespace

Dusty Shapiro · Answer

Same namespace but will give that a shot

Adam Bloom · Answer

also this should be airbyte webapp not airbyte server webapp hosts the API which is used here and in turn connects to server

Adam Bloom · Answer

if same namespace I d try switching to the webapp svc first

Dusty Shapiro · Answer

Ah thanks Adam Will try the full host before just trying the webapp svc

Dusty Shapiro · Answer

Got it much thanks

Dusty Shapiro · Answer

assume port will change to 80 instead of 8001

Dusty Shapiro · Answer

```airbyte airbyte server svc ClusterIP lt none gt 8001 TCP 19d airbyte airbyte webapp svc ClusterIP lt none gt 80 TCP 19d```

Adam Bloom · Answer

correct

Dusty Shapiro · Answer

cool

Dusty Shapiro · Answer

You rock that was exactly it

Adam Bloom · Answer

< ben> I m assuming you re doing most of your work with a docker compose deployment of airbyte Looks like we may want to add a sentence or two to the integration docs to cover k8s deployments of airbyte I can take a stab at that if you aren t as familiar

ben · Answer

Thanks for figuring this out < Adam Bloom>

ben · Answer

I ve been alternatively developing against a local docker compose instance and a k8s deployment but one that s set up behind some ingresses so I haven t encountered this host trickiness I think there s two follow ups here I ll add a note for how to find that connection info depending on how you ve deployed amp make the error messages a bit more clear so if someone else runs into this they get a better sense of what s wrong

Adam Bloom · Answer

yeah my configs look totally different if I m running locally for development against the k8s cluster vs our actual deployments running in the same cluster Sounds familiar slightly smiling face

Dusty Shapiro · Answer

Happy to provide any feedback on my use case and deployment

ben · Answer

I am curious what your deployment set up is right now do you access the web UI through a public IP ingress and then communicate Dagster lt gt Airbyte over Kubernetes DNS That makes sense avoids putting API traffic over the public internet requiring auth but just want to make sure

Dusty Shapiro · Answer

So I know nothing about networking but we override the ingress value in the Helm chart to enable it s use and provide a different ingress class name

Dusty Shapiro · Answer

as well as a google proxy middleware of sorts

Dusty Shapiro · Answer

But yea we have these two annotations with custom values ``<http kubernetes io external dns class|kubernetes io external dns class> `

Dusty Shapiro · Answer

`<http traefik ingress kubernetes io router middlewares|traefik ingress kubernetes io router middlewares> `

Dusty Shapiro · Answer

and then the dagit helm values provide a new host which basically provides a suffix of our company domain

Dusty Shapiro · Answer

This is the first time I ve setup the Airbyte lt gt Dagster connection but we did end up just passing `airbyte airbyte webapp svc` as the AIRBYTE HOST

Adam Bloom · Answer

We have a private ingress VPN only access using istio for ingress routing setup for both our dagster and airbyte deployments in k8s Everything between dagster and airbyte stays internal to our cluster so we use k8s DNS for that