https://dagster.io/ logo
#integration-airbyte
Title
# integration-airbyte
j

James Rosenthal

02/08/2023, 11:28 PM
Hi all! I am running Dagster Serverless and trying to create an Airbyte connection when running Airbyte locally in the docker container. My airbyte_resource is configured as:
Copy code
airbyte_instance = airbyte_resource.configured(
    {
        "host": os.getenv("AIRBYTE_HOST"),
        "port": os.getenv("AIRBYTE_PORT"),
        "username": os.getenv("AIRBYTE_USERNAME"),
        "password": os.getenv("AIRBYTE_PASSWORD"),
    }
)
With host as
"localhost"
, port as
"8000"
, username as
"airbyte"
, and password as
"password"
. From the Airbyte docker, I can see
"POST /api/v1/workspaces/list HTTP/1.1" 401 701 "-" "python-requests/2.28.2"
I keep getting the error:
Request to Airbyte API failed: 401 Client Error: Unauthorized for url: <http://localhost:8000/api/v1/workspaces/list>
Am I getting any of the credentials wrong or are there any settings I need to change in the Dagster UI? Let me know if anyone know of a fix. Thanks!
I also tried (from a thread in the Airbyte Slack):
Copy code
airbyte_instance = airbyte_resource.configured(
    {
        "host": "localhost",
        "port": "8000",
        "request_additional_params": {"auth": ("airbyte", "password")},
    },
)
and I got the same error
d

Dusty Shapiro

02/09/2023, 1:01 AM
Should the port be
5432
?
j

James Rosenthal

02/09/2023, 1:08 AM
5432
is the port I have for Postgres... shouldn't the Airbyte connection use
8000
? I tried using
host.docker.internal
earlier and it did not work either. Let me double check that
d

Dusty Shapiro

02/09/2023, 1:10 AM
You’re right, not sure why I was thinking Postgres port
j

James Rosenthal

02/09/2023, 1:11 AM
I tried both again and I did not have any luck. For more connect, the connection is between a Postgres source and a Snowflake destination. I am not sure if that would have any impact on the outcome
w

Walid El Bouchikhi

02/09/2023, 8:21 AM
Dagster seems to connect to your Airbyte instance. What might not be correct here is the credentials. Make sure your .env file contains the correct credentials on Airbyte and on Dagster.
j

James Rosenthal

02/09/2023, 2:14 PM
That’s right… I can see that the Airbyte terminal getting the request from Dagster. Is there anything I would need in my .env file other than the Airbyte host, post, username, and password? Is there something it should have for Dagster too?
w

Walid El Bouchikhi

02/09/2023, 2:28 PM
When navigating to airbyte on the browser, do you log in as
airbyte/password
? Make sure these environment variables are set in your airbyte container
Copy code
BASIC_AUTH_USERNAME=airbyte
BASIC_AUTH_PASSWORD=password
If you are using docker-compose (with the airbyte repo checked), check the
.env
file. Also, to use an environment variable using dagster for your airbyte instance, try this
Copy code
{
    "host": {"env": "AIRBYTE_HOST"},
    "port": { "env": "AIRBYTE_PORT"},
    "username": { "env": "AIRBYTE_USER"},
    "password" : { "env" : "AIRBYTE_PASSWORD"},
}
Make sure these environment variables are defined in the dagster container (in the
.env
file if you are running dagster locally). I would recommend hard-coding these credentials at first, then retrieving them via the environment variables if that works.
j

James Rosenthal

02/09/2023, 4:41 PM
On the browser (localhost:8000) it does not prompt me for my username or password; it simply loads. I tried setting host and port only, but that did not work for me either
Let me try the
{"env": "CREDENTIAL"}
format
w

Walid El Bouchikhi

02/09/2023, 4:43 PM
Browse the airbyte interface in private mode to see if it requires authentication. Your browser should have stored your authentication.
j

James Rosenthal

02/09/2023, 4:48 PM
Ah, that's a good trick. It did require auth and
airbyte
and
password
worked
w

Walid El Bouchikhi

02/09/2023, 4:49 PM
Ok, next try using hardcoded values for your airbyte instance config on dagster
Copy code
{
    "host": "localhost",
    "port": "8000,
    "username": "airbyte",
    "password" : "password"
}
j

James Rosenthal

02/09/2023, 5:03 PM
Hmmm... I tried both hard coding with:
Copy code
{
  "host": "localhost",
  "port": "8000",
  "username": "airbyte",
  "password": "password"
}
and also setting environment variables with:
Copy code
{
    "host": {"env": "AIRBYTE_HOST"},
    "port": {"env": "AIRBYTE_PORT"},
    "username": {"env": "AIRBYTE_USERNAME"},
    "password" : {"env" : "AIRBYTE_PASSWORD"},
}
and both times I get the same error:
Request to Airbyte API failed: 401 Client Error: Unauthorized for url: <http://localhost:8000/api/v1/workspaces/list>
I also tried restarting both the Dagster cloud UI and the local Airbyte. That did not fix the issue either
Are there some settings I need to set on the Dagster cloud UI first? Or would this run better locally?
w

Walid El Bouchikhi

02/09/2023, 5:04 PM
Are you using Dagster cloud? If so, then it's normal that it could not reach airbyte at
localhost
You'll need to deploy it on the cloud
j

James Rosenthal

02/09/2023, 5:05 PM
Yes, I am using Dagster cloud serverless for this project
My understanding is that we cannot yet configure Connections stored on Airbyte cloud from an interface such as Dagster
Or is there a way to call the Airbyte API from Dagster?
w

Walid El Bouchikhi

02/09/2023, 5:09 PM
Can you try exposing your local airbyte to internet ? You can use ngrok Install it and then run
ngrok http 8000
, it will output an url and a port that you will have to set for your airbyte instance config
j

James Rosenthal

02/09/2023, 5:10 PM
Thanks, let me try that!
And then I want to see
Forwarding          https://{####}.<http://ngrok.io|ngrok.io> -> <http://localhost:8000>
Which one would I need to add to the Airbyte config? Should I be seeing a new port that I need to use?
w

Walid El Bouchikhi

02/09/2023, 5:32 PM
That means port 80
j

James Rosenthal

02/09/2023, 5:33 PM
Let me try that
I got the same error
7 Views