https://dagster.io/ logo
#deployment-ecs
Title
# deployment-ecs
c

Chris Zubak-Skees

03/30/2023, 3:41 PM
I'm using Service Discovery/CloudMap in ECS to make a code location service discoverable, but it's with a network mode that requires a SRV DNS record instead of A. Getting a DNS resolution failed qtype=A failure from Dagster's GRPC. Assume it's not looking for the right DNS record. Does this strike anyone deploying to ECS as familiar?
m

Mike Atlas

03/30/2023, 3:54 PM
the Daemon's going to eventually attempt to resolve DNS at the VPC level (via DHCP option set)
in my team's deployment, we're using a private r53 zone configured by cloudmap. we have our vpc dns set to use
169.254.169.253
(route 53 private resolver)
b

Bethany Yollin

03/30/2023, 7:38 PM
@Chris Zubak-Skees Very familiar! I'm not sure if this is your same issue, but I saw this message a lot when I started playing around with service discovery. I had a lightbulb moment reading the section “Deployment order” in the AWS docs here. Specifically this section:
Existing tasks can't resolve and connect to the new endpoint. Only new Amazon ECS tasks that have a Service Connect configuration in the same namespace and that start running after this deployment can resolve and connect to this endpoint. For example, an Amazon ECS service that runs a client application must start new tasks after the deployment completes of the server that it connects to.
In the end, I had to ensure the gRPC service was running before the daemon and dagit tasks. Not sure if this is your same issue, but the deployment order turned out to be the issue when I received a similar message.
c

Chris Zubak-Skees

03/31/2023, 1:54 AM
Ooooh! Thank you!
Both
I was unfamiliar with Service Connect, but I'm going to try it instead of Service Discovery
6 Views