Hey I m trying to implement dagster with airbyte and dbt but

Question

Hey I m trying to implement dagster with airbyte and dbt but with airbyte and dbt running on separate EC2 instances so would need to SSH into it do you have an example where something similar has been implemented I found this <https docs dagster io apidocs libraries dagster ssh> but it doesn t have a lot of explanation and I imagine SSH is a pretty common use case

chris · Answer

Have you checked out the <https docs dagster io apidocs libraries dagster airbyte|dagster airbyte> integration It allows specifying a port and host for communicating with the airbyte API so presumably you could just point it at your airbyte ec2 instance Then you could just run everything from the ec2 instance that s running dbt

Isaac Harris-Holt · Answer

We have a load balancer pointing at our Airbyte instance anyway and both the Airbyte instance and Dagster instance have static IPs so we just allowed the Dagster instance ingress through the load balancer

Konrad Schlatte · Answer

Yes I ve seen the docs but I am not clear on how I can configure the airbyteResource if I use SSH as all there is available is host and port so host will be localhost and port 8000 but that will only work if I m already in the correct instance

Konrad Schlatte · Answer

there s a ssh library as well but not sure how that would work together with airbyte implementation

chris · Answer

You wouldn t connect to the ec2 instance via SSH you would communicate with the airbyte API that you ve stood up via HTTP

chris · Answer

If you need to communicate to the ec2 instance via ssh then you would need to use dagster ssh I don t think we have any specific examples of doing this with airbyte in particular Good call out though on general lack of documentation on dagster ssh <https github com dagster io dagster issues 8746|made an issue >to add some examples here

Konrad Schlatte · Answer

yes well if airbyte is running on a different server I do need to connect to it but it is interesting that this seems to be unusual Would you generally expect for dagster to run on the same server as the services it orchestrates My use case is fairly simple i e 1 run airbyte ingest data into database 2 run dbt model From your experience would you expect that all to happen on the same server

chris · Answer

definitely would not expect that to happen all on the same server I think we generally expect people to connect to these services via some sort of an API that has a more robust connection framework So it s the SSH piece here that s a little bit unique

Isaac Harris-Holt · Answer

The source code itself is pretty self documenting so I found my way around the library that way but I agree a lot of the integrations are fairly poorly documented on the site