19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hey I'm trying to implement dagster with airbyte a...
# ask-communityk
Konrad Schlatte
07/01/2022, 4:29 PMHey I'm trying to implement dagster with airbyte and dbt but with airbyte and dbt running on separate EC2 instances - so would need to SSH into it, do you have an example where something similar has been implemented? I found this https://docs.dagster.io/_apidocs/libraries/dagster-ssh but it doesn't have a lot of explanation and I imagine SSH is a pretty common use case?
c
chris
07/01/2022, 8:18 PMHave you checked out the dagster airbyte integration? It allows specifying a port and host for communicating with the airbyte API, so presumably you could just point it at your airbyte ec2 instance. Then you could just run everything from the ec2 instance that's running dbt
i
Isaac Harris-Holt
07/04/2022, 7:26 AMWe have a load balancer pointing at our Airbyte instance anyway, and both the Airbyte instance and Dagster instance have static IPs, so we just allowed the Dagster instance ingress through the load balancer
k
Konrad Schlatte
07/05/2022, 1:52 PMYes I've seen the docs but I am not clear on how I can configure the airbyteResource if I use SSH
as all there is available is host and port so
host will be localhost and port 8000 but that will only work if I'm already in the correct instance
Konrad Schlatte
07/05/2022, 2:06 PMthere's a ssh "library" as well but not sure how that would work together with airbyte implementation
c
chris
07/05/2022, 7:46 PMYou wouldn't connect to the ec2 instance via SSH, you would communicate with the airbyte API that you've stood up via HTTP
chris
07/05/2022, 7:53 PMIf you need to communicate to the ec2 instance via ssh, then you would need to use dagster-ssh. I don't think we have any specific examples of doing this with airbyte in particular. Good call-out though on general lack of documentation on dagster-ssh, made an issue to add some examples here.
k
Konrad Schlatte
07/06/2022, 10:32 AMyes well if airbyte is running on a different server I do need to connect to it but it is interesting that this seems to be unusual?
Would you generally expect for dagster to run on the same server as the services it orchestrates? My use case is fairly simple i.e.
1. run airbyte (ingest data into database)
2. run dbt model
From your experience would you expect that all to happen on the same server?
c
chris
07/06/2022, 5:11 PMdefinitely would not expect that to happen all on the same server - I think we generally expect people to connect to these services via some sort of an API that has a more robust connection framework. So it's the SSH piece here that's a little bit unique
i
Isaac Harris-Holt
07/07/2022, 1:58 PMThe source code itself is pretty self-documenting, so I found my way around the library that way, but I agree a lot of the integrations are fairly poorly documented on the site
👍 1