Hi all, question about launching in ECS. I have two containers launched, one running dagit and the other the daemon service. I am able to run my jobs successfully when ssh into the dagit container when the daemon service is not connected (as if it is not up and running). But, when ever the daemon is activated, I keep receiving the error

docker.errors.ImageNotFound: 404 Client Error

and

Not Found ("pull access denied for dagit, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

. I have set the

DAGSTER_CURRENT_IMAGE

to dagit (the name of the ecr repo containing my dagit image). I have also given the task full IAM access to that ecr repo. Any one have any idea how to fix this? Any help would be greatly appreciated. Thanks!

Hi Dominick - could you clarify where exactly you're seeing the error?

Hi @daniel - I am seeing this error after attempting to run the job, the error message is located in the health tab, under run queue status.

Got it - I think DAGSTER_CURRENT_IMAGE usually includes the registry URL as well - something like 764506304434.dkr.ecr.us-west-2.amazonaws.com/dagster-cloud-user-code:demo

For some reason, Dagster is now throwing the error when running the job that certain env variables are not set, while they are being passed into the task as env variables. Also, this was never an issue when running the job without the Daemon active - so just curious as to why the environment variables are behaving differently

Did you switch to using the EcsRunLauncher as part of this change, or were you using the EcsRunLauncher before back when it was working?

Oh, that was most likely my issue. I had the docker run launcher still set from before. Switching over and trying now

once you have the daemon enabled, the runs are going to be launched from the daemon service rather than the dagit service - so it could be that there are missing environment variables on the daemon service that are present on the dagit service?

So I have same set of env variables going to both dagit and daemon task containers. I am now getting this error when trying to run the job with the daemon active:

botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.

Are you running fargate or ec2?

We're on fargate

Ok, I am using ec2. Is this not possible with ec2? Also, strange that I am getting an error for using fargate when I am not?

It's not currently possible but we'd like to support it in the future. I filed a tracking issue here: https://github.com/dagster-io/dagster/issues/6565

When switching to Fargate, I receive this error when trying to deploy the ecs service with “DAEMON” scheduling of a “FARGATE” launch type:

The daemon scheduling strategy does not support the FARGATE launch type. Change the launch type and try again

I think that's an AWS restriction, not a Dagster restriction

How were you able to deploy in aws fargate with Daemon?

I didn't think that we were - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html says "Fargate tasks do not support the 

DAEMON

 scheduling strategy." Is that different than what you're seeing?

I guess its just a little confusing. You are launching the dagit and dagster-daemon tasks with the replica scheduling strategy, not Daemon?

I believe so

We use the docker-compose integration with ECS which I believe runs everything with the REPLICA strategy https://github.com/docker/compose-cli/blob/eda269b742e55b4d8c09d8b5f21e7b0ee2940ff5/ecs/cloudformation.go#L270