Hello all. I'm trying to run dagster on ECS in one account (Account B) in an existing VPC that belongs to another account in my organization (Account A). I am using the deploy_ecs example as a base (https://github.com/dagster-io/dagster/tree/master/examples/deploy_ecs). During deployment, if I have specified the VPC that does not belong to the account in which dagster is being deployed, the deployment fails as cross-account service discovery is not implemented in AWS (according to my organization's contact at AWS). Has anyone deployed dagster in a shared VPC from another account? I am trying to avoid creating a VPC for dagster in Account B and peering it with the VPC in Account A if possible. Thanks for your input and suggestions.

@Mark Durbin Not sure about ECS service-discovery specifically, but I've successfully used something called VPC Private Link endpoints to avoid entire VPC peering/networking

@Mike Atlas thank you for your reply. The only issue with that approach is that we don't allow the users of Account B to access Account A (Account A is a shared services account, and Account B is used by one of our "customers" so-to-speak"; in an ideal world, the users of Account B can deploy dagster in their account by themselves. My apologies for not clarifying that relationship in my post.

Yeah again, VPC private link would be useful - Account B: full ECS deployment in its own VPC-b Account A: has a shared VPC-a Create the VPC PrivateLink endpoint in account B, share it with Account A instead; now any service in the shared VPC-a should be able to resolve the PrivateLink endpoint over to Dagster in AccountB-VPC-b

Thanks again @Mike Atlas. I was hoping to avoid creating and managing the second VPC but Private Link seems like a better option than peering.

Yeah np, this similar to a pattern we are using at my company. There's a "production" shared VPC for most services in a managed AWS account ("Account A/VPC-a") Then there's Dagster, which is running in a dedicated AWS account B+VPC-b, because it's processing HIPAA data