Hi all! I am running Dagster Serverless and trying to create an Airbyte connection when running Airbyte locally in the docker container. My airbyte_resource is configured as:

airbyte_instance = airbyte_resource.configured(
    {
        "host": os.getenv("AIRBYTE_HOST"),
        "port": os.getenv("AIRBYTE_PORT"),
        "username": os.getenv("AIRBYTE_USERNAME"),
        "password": os.getenv("AIRBYTE_PASSWORD"),
    }
)

With host as

"localhost"

, port as

"8000"

, username as

"airbyte"

, and password as

"password"

. From the Airbyte docker, I can see

"POST /api/v1/workspaces/list HTTP/1.1" 401 701 "-" "python-requests/2.28.2"

I keep getting the error:

Request to Airbyte API failed: 401 Client Error: Unauthorized for url: <http://localhost:8000/api/v1/workspaces/list>

Am I getting any of the credentials wrong or are there any settings I need to change in the Dagster UI? Let me know if anyone know of a fix. Thanks!

Should the port be

5432

?

5432

is the port I have for Postgres... shouldn't the Airbyte connection use

8000

? I tried using

host.docker.internal

earlier and it did not work either. Let me double check that

You’re right, not sure why I was thinking Postgres port

I tried both again and I did not have any luck. For more connect, the connection is between a Postgres source and a Snowflake destination. I am not sure if that would have any impact on the outcome

Dagster seems to connect to your Airbyte instance. What might not be correct here is the credentials. Make sure your .env file contains the correct credentials on Airbyte and on Dagster.

That’s right… I can see that the Airbyte terminal getting the request from Dagster. Is there anything I would need in my .env file other than the Airbyte host, post, username, and password? Is there something it should have for Dagster too?

When navigating to airbyte on the browser, do you log in as

airbyte/password

? Make sure these environment variables are set in your airbyte container

BASIC_AUTH_USERNAME=airbyte
BASIC_AUTH_PASSWORD=password

If you are using docker-compose (with the airbyte repo checked), check the

.env

file. Also, to use an environment variable using dagster for your airbyte instance, try this

{
    "host": {"env": "AIRBYTE_HOST"},
    "port": { "env": "AIRBYTE_PORT"},
    "username": { "env": "AIRBYTE_USER"},
    "password" : { "env" : "AIRBYTE_PASSWORD"},
}

Make sure these environment variables are defined in the dagster container (in the

.env

file if you are running dagster locally). I would recommend hard-coding these credentials at first, then retrieving them via the environment variables if that works.

On the browser (localhost:8000) it does not prompt me for my username or password; it simply loads. I tried setting host and port only, but that did not work for me either

Browse the airbyte interface in private mode to see if it requires authentication. Your browser should have stored your authentication.

Ah, that's a good trick. It did require auth and

airbyte

and

password

worked

Ok, next try using hardcoded values for your airbyte instance config on dagster

{
    "host": "localhost",
    "port": "8000,
    "username": "airbyte",
    "password" : "password"
}

Hmmm... I tried both hard coding with:

{
  "host": "localhost",
  "port": "8000",
  "username": "airbyte",
  "password": "password"
}

and also setting environment variables with:

{
    "host": {"env": "AIRBYTE_HOST"},
    "port": {"env": "AIRBYTE_PORT"},
    "username": {"env": "AIRBYTE_USERNAME"},
    "password" : {"env" : "AIRBYTE_PASSWORD"},
}

and both times I get the same error:

Request to Airbyte API failed: 401 Client Error: Unauthorized for url: <http://localhost:8000/api/v1/workspaces/list>

Are you using Dagster cloud? If so, then it's normal that it could not reach airbyte at

localhost

You'll need to deploy it on the cloud

Yes, I am using Dagster cloud serverless for this project

Can you try exposing your local airbyte to internet ? You can use ngrok Install it and then run

ngrok http 8000

, it will output an url and a port that you will have to set for your airbyte instance config

Thanks, let me try that!

That means port 80

Let me try that