Hello, I was wondering if there were best practices around running dagit and user code on AWS ECS. So the question I have is if I should separate the container for dagit and user code into 2 services or can should I those in the same task definition? It seems like keeping the in the same task definition would be the way to go but I'm interested if there are some guidelines about that.

Not saying it’s a best practice, but I’ve deployed dagster on ECS in two companies and both followed the pattern of a single service for dagit+daemon (you could split it if you want to, i didn’t think it necessary) and one service for each use code location, communication between them done through cloudmap/servicediscovery

Thank you. How does the communication work then, you wildcard the host it in the workspace.yaml and then it fetches all user code repositories?

ServiceDiscovery requires a service discovery private dns namespace (which creates a private hosted zone in route53 in the background) and takes care of the networking part, requirement is that they’re all inside the same VPC, otherwise you’d need a little more config, here’s my example

workspace.yaml

,

repo.dagster

is my (private) hosted zone name, and

my-host

is the service discovery service.

load_from:
  - grpc_server:
      host: my-host.repo.dagster
      port: 4266
      location_name: my_cool_location

here’s a terraform snippet for the service discovery service:

resource "aws_service_discovery_private_dns_namespace" "dagster_repo" {
  name = "repo.dagster"
  vpc  = module.vpc.vpc_id
}

resource "aws_service_discovery_service" "this" {
  name = var.service_discovery_name

  dns_config {
    namespace_id = aws_service_discovery_private_dns_namespace.dagster_repo.id

    dns_records {
      ttl  = 300
      type = "A"
    }

    routing_policy = "MULTIVALUE"
  }

  health_check_custom_config {
    failure_threshold = 1
  }
}