Hello all. I'm trying to run dagster on ECS in one account (Account B) in an existing VPC that belongs to another account in my organization (Account A). I am using the deploy_ecs example as a base (https://github.com/dagster-io/dagster/tree/master/examples/deploy_ecs). During deployment, if I have specified the VPC that does not belong to the account in which dagster is being deployed, the deployment fails as cross-account service discovery is not implemented in AWS (according to my organization's contact at AWS). Has anyone deployed dagster in a shared VPC from another account? I am trying to avoid creating a VPC for dagster in Account B and peering it with the VPC in Account A if possible. Thanks for your input and suggestions.
12/12/2022, 4:48 PM
@Mark Durbin Not sure about ECS service-discovery specifically, but I've successfully used something called VPC Private Link endpoints to avoid entire VPC peering/networking
I think you could probably create your Dagster service in Account A; put an NLB in front of it; then make a VPC Private Link endpoint through to Account B
12/12/2022, 4:57 PM
@Mike Atlas thank you for your reply. The only issue with that approach is that we don't allow the users of Account B to access Account A (Account A is a shared services account, and Account B is used by one of our "customers" so-to-speak"; in an ideal world, the users of Account B can deploy dagster in their account by themselves. My apologies for not clarifying that relationship in my post.
12/12/2022, 5:07 PM
Yeah again, VPC private link would be useful -
Account B: full ECS deployment in its own VPC-b
Account A: has a shared VPC-a
Create the VPC PrivateLink endpoint in account B, share it with Account A instead; now any service in the shared VPC-a should be able to resolve the PrivateLink endpoint over to Dagster in AccountB-VPC-b
one small caveat: VPC-b has to live in the same region and AZ as VPC-a
Thanks again @Mike Atlas. I was hoping to avoid creating and managing the second VPC but Private Link seems like a better option than peering.
12/12/2022, 8:17 PM
Yeah np, this similar to a pattern we are using at my company.
There's a "production" shared VPC for most services in a managed AWS account ("Account A/VPC-a")
Then there's Dagster, which is running in a dedicated AWS account B+VPC-b, because it's processing HIPAA data
We use a Private Link to ship sanitized data back to AccountA
(without VPC peering and without going over the wide internet)