https://dagster.io/ logo
Powered by
#integration-airbyte
Title
# integration-airbyte
l

Lindsay S

01/24/2023, 9:30 PM
Any ideas what I could be doing wrong? I am trying out the example. I am running dagster in docker locally and just started airbyte locally, too. I created my first connection that is syncing now. • dagster-airbyte version 0.17.10. • defaults only • all code is in repository.py • when I comment out this repository, the other repositories do load
Copy code
from dagster_airbyte import airbyte_resource, load_assets_from_airbyte_instance

airbyte_instance = airbyte_resource.configured(
    {
        "host": "localhost",
        "port": "8000",
        # If using basic auth, include username and password:
        "username": "airbyte",
        "password": "password",
    }
)

# Use the airbyte_instance resource we defined in Step 1
airbyte_assets = load_assets_from_airbyte_instance(airbyte_instance)

@repository
def my_repo():
    return [airbyte_assets,]
b

ben

01/24/2023, 9:55 PM
Hi Lindsay, just to make sure: • You’ve kicked off a sync from the Airbyte UI • You’re trying to load 
repository.py
in Dagster at the same time • The repository load is timing out / not loading at all
l

Lindsay S

01/24/2023, 9:55 PM
yes. that's right. the only thing I can think is maybe it doesn't like that it's syncing and trying to load?
b

ben

01/24/2023, 9:56 PM
It’s possible - let me see if I can replicate the issue
l

Lindsay S

01/24/2023, 9:56 PM
i just really want to see how load assets works. i don't really have much to connect right now
b

ben

01/24/2023, 10:02 PM
You mentioned you are running Dagster in Docker - you might be able to get an error message from the user code server (which it looks like Dagster is unable to reach from the error message)
It’s possible this may be a Docker networking issue
l

Lindsay S

01/24/2023, 10:05 PM
yeh, I wondered that. ok. let me see if I can test that out
b

ben

01/24/2023, 10:09 PM
It’s also possible that if the container is resource-constrained when loading from the instance and is running out of memory - in either case the Docker logs should give us more information
l

Lindsay S

01/24/2023, 10:51 PM
ok. I waited for that sync to complete and retried - still no luck. but definitely is a networking issue that I assume is due to not not listening inside the container. here is the error:
Copy code
dagster_code_runner |   airbyte_assets = load_assets_from_airbyte_instance(airbyte_instance)
dagster_code_runner | 2023-01-24 22:43:04 +0000 - dagster - ERROR - system - Request to Airbyte API failed: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v1/workspaces/list (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff7443d250>: Failed to establish a new connection: [Errno 111] Connection refused'))
dagster_code_runner | 2023-01-24 22:43:05 +0000 - dagster - ERROR - system - Request to Airbyte API failed: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v1/workspaces/list (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff7449f8d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
dagster_code_runner | 2023-01-24 22:43:05 +0000 - dagster - ERROR - system - Request to Airbyte API failed: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v1/workspaces/list (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff74454150>: Failed to establish a new connection: [Errno 111] Connection refused'))
dagster_code_runner | 2023-01-24 22:43:05 +0000 - dagster - ERROR - system - Request to Airbyte API failed: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v1/workspaces/list (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff7445c250>: Failed to establish a new connection: [Errno 111] Connection refused'))
I added 
extra_hosts:
- "host.docker.internal:host-gateway"
to my docker-compose.yaml for the user_code container with no luck. i'll keep trying but wonder if there are any examples you could suggest
b

ben

01/24/2023, 11:54 PM
Hmm, yeah, it looks like the user code server can’t see/reach the Airbyte instance. I’ll spin up both in docker-compose and see if I can get the same networking issue.
My guess is we could either put them on the same docker-compose network or broker through the host network (e.g. using 
host.docker.internal
instead of 
localhost
)
I ran into the same issue locally running Dagster + Airbyte through Docker, and changing the Airbyte host from 
localhost
to 
host.docker.internal
fixed it, hopefully that works for you as well
l

Lindsay S

01/25/2023, 12:04 AM
did you change it in airbyte's docker-compose file?
b

ben

01/25/2023, 12:30 AM
In the Airbyte resource configuration in my Dagster code
l

Lindsay S

01/25/2023, 3:21 AM
that worked. thanks! it would have taken me a long time to figure that one out.
2 Views
Powered by