Hello, has anyone correctly set up an ECS deployme...
# deployment-ecs
Hello, has anyone correctly set up an ECS deployment with a specific image tag for a pipelines task/service? The task that is created for the dagster-run seems to ignore the image tag and just default to "latest". For example, I have the
variable set to
with an image tag tied to a commit hash, which loads correctly for the pipelines task/service, but is ignored for the dagster-run. Thanks!
Are you managing your own deployment of ECS or using the docker compose example in the docs? https://github.com/dagster-io/dagster/tree/master/examples/deploy_ecs Can you double check that you’re indeed setting
in the pipelines service? In the example it defaults to latest https://github.com/dagster-io/dagster/blob/7ae9dffc389044690742a4bea19fbd6c820e7835/examples/deploy_ecs/docker-compose.yml#L103
The run launcher is sourcing the image from the repository origin: https://github.com/dagster-io/dagster/blob/7ae9dffc389044690742a4bea19fbd6c820e7835/python_modules/libraries/dagster-aws/dagster_aws/ecs/launcher.py#L94 Which is in turn sourcing the image from that envvar.
I modified the docker compose example. It works perfectly, except the run launched uses the latest tag
Copy code
# This service runs a gRPC server that serves information about your
# repository. By setting DAGSTER_CURRENT_IMAGE to its own image, we tell the
# run launcher to use this same image when launching runs in a new container.
# Multiple containers like this can be deployed separately - each needs to
# run on its own port and have its own entry in the workspace.yaml file.
    context: .
    dockerfile: ./Dockerfile
    target: pipelines
  image: "$REGISTRY_URL/<my-ecr-registry>/pipelines:$DAGSTER_IMAGE_TAG"
  container_name: pipelines
  command: "dagster api grpc -h -p 4000 -f dagster_repository.py"
    DAGSTER_CURRENT_IMAGE: "$REGISTRY_URL/<my-ecr-repository>/pipelines:$DAGSTER_IMAGE_TAG"
this is the pipelines section I have set up
In the task definition on ECS, I can see that the
was set correctly to:
Copy code
  "value": "<REGISTRY_URL>/<my-ecr-repository>/pipelines:b4ea218de98329ab95d322190861d5fc2c82deeb"
so the tag made it there, but then the dagster-run task has no tag associated with it when I try to launch a run from the dagit UI
spinthinking 1
You know what, let me try rebuilding one more time. I'm wondering if there's an image somewhere with an old value of that environment variable.
I’m trying to reproduce now but yeah, that’d be helpful.
👍 1
It appears that was it, or CloudFormation just took too long to update. So sorry to waste your time!
No worries! Glad to hear it’s working.
You all rock 👍