https://dagster.io/ logo
#deployment-ecs
Title
# deployment-ecs
d

Dominick Giordano

02/10/2022, 8:38 PM
Hi all, question about launching in ECS. I have two containers launched, one running dagit and the other the daemon service. I am able to run my jobs successfully when ssh into the dagit container when the daemon service is not connected (as if it is not up and running). But, when ever the daemon is activated, I keep receiving the error
docker.errors.ImageNotFound: 404 Client Error
and
Not Found ("pull access denied for dagit, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
. I have set the
DAGSTER_CURRENT_IMAGE
to dagit (the name of the ecr repo containing my dagit image). I have also given the task full IAM access to that ecr repo. Any one have any idea how to fix this? Any help would be greatly appreciated. Thanks!
d

daniel

02/10/2022, 11:46 PM
Hi Dominick - could you clarify where exactly you're seeing the error?
d

Dominick Giordano

02/11/2022, 5:30 AM
Hi @daniel - I am seeing this error after attempting to run the job, the error message is located in the health tab, under run queue status.
d

daniel

02/11/2022, 1:17 PM
Got it - I think DAGSTER_CURRENT_IMAGE usually includes the registry URL as well - something like 764506304434.dkr.ecr.us-west-2.amazonaws.com/dagster-cloud-user-code:demo
so registry/image:tag
1
d

Dominick Giordano

02/11/2022, 3:40 PM
For some reason, Dagster is now throwing the error when running the job that certain env variables are not set, while they are being passed into the task as env variables. Also, this was never an issue when running the job without the Daemon active - so just curious as to why the environment variables are behaving differently
d

daniel

02/11/2022, 3:43 PM
Did you switch to using the EcsRunLauncher as part of this change, or were you using the EcsRunLauncher before back when it was working?
d

Dominick Giordano

02/11/2022, 3:48 PM
Oh, that was most likely my issue. I had the docker run launcher still set from before. Switching over and trying now
d

daniel

02/11/2022, 3:49 PM
once you have the daemon enabled, the runs are going to be launched from the daemon service rather than the dagit service - so it could be that there are missing environment variables on the daemon service that are present on the dagit service?
d

Dominick Giordano

02/11/2022, 5:24 PM
So I have same set of env variables going to both dagit and daemon task containers. I am now getting this error when trying to run the job with the daemon active:
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.
Are you running fargate or ec2?
d

daniel

02/11/2022, 5:27 PM
We're on fargate
d

Dominick Giordano

02/11/2022, 5:30 PM
Ok, I am using ec2. Is this not possible with ec2? Also, strange that I am getting an error for using fargate when I am not?
d

daniel

02/11/2022, 6:13 PM
It's not currently possible but we'd like to support it in the future. I filed a tracking issue here: https://github.com/dagster-io/dagster/issues/6565
d

Dominick Giordano

02/11/2022, 6:48 PM
When switching to Fargate, I receive this error when trying to deploy the ecs service with “DAEMON” scheduling of a “FARGATE” launch type:
The daemon scheduling strategy does not support the FARGATE launch type. Change the launch type and try again
d

daniel

02/11/2022, 7:03 PM
I think that's an AWS restriction, not a Dagster restriction
d

Dominick Giordano

02/11/2022, 7:09 PM
How were you able to deploy in aws fargate with Daemon?
d

daniel

02/11/2022, 7:11 PM
I didn't think that we were - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html says "Fargate tasks do not support the 
DAEMON
 scheduling strategy." Is that different than what you're seeing?
d

Dominick Giordano

02/11/2022, 7:13 PM
I guess its just a little confusing. You are launching the dagit and dagster-daemon tasks with the replica scheduling strategy, not Daemon?
d

daniel

02/11/2022, 7:15 PM
I believe so
j

jordan

02/11/2022, 8:06 PM
We use the docker-compose integration with ECS which I believe runs everything with the REPLICA strategy https://github.com/docker/compose-cli/blob/eda269b742e55b4d8c09d8b5f21e7b0ee2940ff5/ecs/cloudformation.go#L270
8 Views