https://dagster.io/ logo
Title
m

Marcin Grzybowski

11/17/2022, 10:05 AM
hello, I have 3 dagster projects in 3 different github repositories. I am creating Docker Images (with app code) and I am uploading them after to ECR via GitHub Actions (+ecs deploy). I keep ECS_TASK configuration in Terraform This setup is working with regular
dagster.core.launcher
but I would like to use dagster_aws.ecs (EcsRunLauncher) to decrease amount of AWS Fargate instances and launch them when I need them (via Dagster UI/Scheduler). Is it possible to configure
EcsRunLauncher
for each of different AWS ECS Tasks?
For example: 
 APP1    -> ecs_task_app1 (Docker_Image_app1)

 APP2   ->  ecs_task_app2 (Docker_Image_app2)
 APP2_1 ->  ecs_task_app2 (Docker_Image_app2)
 APP2_2 ->  ecs_task_app2 (Docker_Image_app2)

 APP3   ->  ecs_task_app3 (Docker_Image_app3)
d

daniel

11/17/2022, 2:33 PM
Hi Marcin - you can run a separate service for each task by creating a separate repository location for each one like in the github example linked here: https://docs.dagster.io/deployment/guides/aws#deploying-in-ecs and you can vary the task definition that's used for runs launched from each of the different repository locations using the method here: https://dagster.slack.com/archives/C014UDS8LAV/p1667921690467969
m

Marcin Grzybowski

11/17/2022, 2:41 PM
Hello, I used this config:
run_launcher:
    module: dagster_aws.ecs
    class: EcsRunLauncher
    config:
      task_definition: "arn:aws:ecs:us-east-1:XXXXXXXXXXXXX:task-definition/dev-dagster-project1:2"       
      container_name: "dev_dagster_project1"
and I ran project based on this Task Definition. I am not sure where I should overwrite this configuration to use different Task Definition in other job()
d

daniel

11/17/2022, 2:43 PM
What we recommend doing is running a grpc server for each of your tasks like in the deploy_ecs example here: https://github.com/dagster-io/dagster/tree/1.0.17/examples/deploy_ecs Each of those services can then be deployed separately from dagit and serve information that tells the run launcher what image or task definition to use
m

Marcin Grzybowski

11/17/2022, 2:51 PM
ok, Thank you for the hints and help!