https://dagster.io/ logo
Powered by
#dagster-ecs
Title
# dagster-ecs
l

Lindsay S

01/04/2023, 7:42 PM
Hi everyone, I am still having the same issue with "A load balancer cannot be attached to multiple subnets in the same Availability Zone (Service: AmazonElasticLoadBalancing; Status Code: 400; Error Code: InvalidConfigurationRequest; Request ID: <id redacted>; Proxy: null)" when trying to deploy. I am trying with the example code. I was wondering if someone could help me understand how the load balancer is created and/or point me to source for the ecs run launcher module. Since I am just using the example code I am confused about what could be happening. there is no load balancer created as far as I can see
m

Mike Atlas

01/04/2023, 7:56 PM
I'm not using the docker-compose example myself, but am using ECS, and an ALB is needed to route http requests to the Dagit ECS task/service
l

Lindsay S

01/04/2023, 7:56 PM
I think that the module creates a load balancer if it doesn't exist
I wanted to see how that is done
m

Mike Atlas

01/04/2023, 7:59 PM
I'm guessing your VPC has multiple subnets defined for the same AZ, and the script here is trying (naively) to associate to all available subnets in the VPC
I suppose you could try making a dedicated VPC for running Dagster in ECS and only one subnet for it
The error messages and ALB are created for you by docker-compose, not by Dagster
https://github.com/dagster-io/dagster/tree/master/examples/deploy_ecs#deploying-dagster
One of the resources Docker creates is an AWS ELB Load Balancer. You can access Dagit on port 3000 of the Load Balancer's URL. To find the Load Balancer's URL, look for the Load Balancer tagged with our project name.
l

Lindsay S

01/04/2023, 8:08 PM
I see. yes, i'm looking at the subnets for the default vpc now and I see the issue. I will try to specify the subnets in the docker-compose.
🎉 1
if I did want to create another vpc for just running dagster, where would I set the vpc I want to use? also in the docker-compose file?
m

Mike Atlas

01/04/2023, 8:11 PM
yeah - looks like 
x-aws-vpc
is the key?
l

Lindsay S

01/04/2023, 8:12 PM
thank you so much. trying to set subnets for the default first and if that doesn't work I will try creating a new vpc. I don't want to mess with the current vpc because I have no idea what is using that. this is really helpful, though
👍 1
m

Mike Atlas

01/04/2023, 8:13 PM
Yeah. That docker link ^ gave me another idea, which is that you could define an ALB that you made separately and just refer to it with 
x-aws-loadbalancer
that might make it possible to stay in the same VPC
but you can also specify ALB subnets it seems anyways, another solution is the section in https://github.com/docker/compose-cli/issues/921#issuecomment-998704609
Copy code
LoadBalancer:
      Properties:
        Subnets:
        - subnet-xxxxxxxxx
So maybe you can just pick 1 subnet from your default/main VPC and be all set ^
l

Lindsay S

01/04/2023, 8:40 PM
yaaasss! this worked. Thank you so much. I am still getting errors deploying, but they are different errors, so I think this piece, at least, is solved. I ended up using the last solution, adding something like this to the end of my docker-compose file
🌈 1
Copy code
x-aws-cloudformation:
  Resources:
    LoadBalancer:
      Properties:
        Subnets:
        - subnet-xxxxxxx
        - subnet-xxxxxxx
catjam 1
m

Mike Atlas

01/04/2023, 9:41 PM
Awesome!
9 Views
Powered by