https://dagster.io/ logo
#deployment-ecs
Title
# deployment-ecs
o

Osman Mesut Ozcan

02/08/2023, 2:10 PM
I am trying to setup an ECS pipeline with a couple of linear jobs. Some of these jobs need to run on FARGATE and some of them are in EC2. Curious, if this is something supported by default? I figured I cannot override the
launchType
through @job decorator. I can define it in the
run_launcher
in
dagster.yaml
but as far as I understand we cannot provide multiple run_launcher settings through it. Anyone have any experience with this? TLDR; Can we use different ECS launch types for different `@job`s within the same definition?
a

Andrew Grigorev

02/08/2023, 3:17 PM
you can specify task_arn via
ecs/task_arn
task, and specify
requiresCompatibilities
in corresponding task definition
hmm, sorry, nope, its not the task_definition_arn
it just a computed arn of the task itself 😕
o

Osman Mesut Ozcan

02/08/2023, 3:19 PM
Yeah I fell for that as well 😄
a

Andrew Grigorev

02/08/2023, 3:20 PM
so probably the answer would be "no") you could only achieve this with a separate code locations
d

daniel

02/08/2023, 3:25 PM
The answer is currently no - if this would be useful for you would it be possible to file a feature request? Would probably involve adding some kind of tag functionality on individual jobs that let you customize the run_task arguments
o

Osman Mesut Ozcan

02/08/2023, 3:26 PM
Can do a feature request 👍 I am wondering if there is a problem with my understanding of
job
. To elaborate, here is a simple graph of what I am trying to put together.
Copy code
S3 file sensor ---> asset 1: CPU bound ---> asset 2: GPU bound ---> uploads result to S3
and I am thinking each of these asset steps as separate jobs that I want to run on different ECS resources
would love to know if there is more idiomatic way of doing this
d

daniel

02/08/2023, 3:27 PM
If it would be more natural to express those as ops in the same job then this other feature request may be relevant as well (an executor that lets you run each op/asset in its own ECS task): https://github.com/dagster-io/dagster/issues/9671
The ECS integration is a bit behind the k8s integration in terms of customizability features like these currently
We do support setting different resource requirements for jobs in ECS, but we don't yet support varying fargate vs. EC2
o

Osman Mesut Ozcan

02/08/2023, 3:32 PM
I understand. I saw the memory/cpu bits. launchType is somewhat of a relaxed requirement but we need a GPU setting at the very minimum. In that case, I will open a discussion with my team about potentially going for k8s version
d

daniel

02/08/2023, 3:32 PM
We also support setting task overrides for a particular job via tags: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_TaskOverride.html (but Ec2 isn't in that list unfortunately)
17 Views