I am trying to setup an ECS pipeline with a couple of linear jobs. Some of these jobs need to run on FARGATE and some of them are in EC2. Curious, if this is something supported by default? I figured I cannot override the

launchType

through @job decorator. I can define it in the

run_launcher

in

dagster.yaml

but as far as I understand we cannot provide multiple run_launcher settings through it. Anyone have any experience with this? TLDR; Can we use different ECS launch types for different `@job`s within the same definition?

you can specify task_arn via

ecs/task_arn

task, and specify

requiresCompatibilities

in corresponding task definition

Yeah I fell for that as well 😄

so probably the answer would be "no") you could only achieve this with a separate code locations

The answer is currently no - if this would be useful for you would it be possible to file a feature request? Would probably involve adding some kind of tag functionality on individual jobs that let you customize the run_task arguments

Can do a feature request 👍 I am wondering if there is a problem with my understanding of

job

. To elaborate, here is a simple graph of what I am trying to put together.

S3 file sensor ---> asset 1: CPU bound ---> asset 2: GPU bound ---> uploads result to S3

and I am thinking each of these asset steps as separate jobs that I want to run on different ECS resources

If it would be more natural to express those as ops in the same job then this other feature request may be relevant as well (an executor that lets you run each op/asset in its own ECS task): https://github.com/dagster-io/dagster/issues/9671

I understand. I saw the memory/cpu bits. launchType is somewhat of a relaxed requirement but we need a GPU setting at the very minimum. In that case, I will open a discussion with my team about potentially going for k8s version

We also support setting task overrides for a particular job via tags: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_TaskOverride.html (but Ec2 isn't in that list unfortunately)