Hey 👋 I’m looking for advice on which deployment approach I should look at before I invest in learning about ECS or K8s Keep in mind this is all new territory for me. We are starting to build a forecasting framework around dagster which uses a combination of DBT and python and will run on AWS. The DBT steps are for pre-processing / post-processing and obviously don’t need a lot of resources. We also have python ops/ assets downstream that will have various resource requirements. At a minimum we will want probably X1 instances for loading large datasets and doing some light work with multiprocessing. Then we have a model training op / asset which will need a GPU. Given that I will need different instances for each op in the compute graph, what are my options for deployment? The ECS deployment tutorial seems to be doable with my level of knowledge but there’s nothing in there about specific compute resources per op (from what I can see) - any resources on this would be greatly appreciated. P.s. I use ops / assets above because I have used ops in the past but will try and use the asset based approach going forward if there is no reason not to

You can customize the compute resources on a job basis in ECS: https://docs.dagster.io/deployment/guides/aws#customizing-cpu-memory-and-ephemeral-storage-in-ecs As far as the choice between K8s/ECS is concerned, I don’t feel comfortable giving advice. ECS is much simpler and easier to pick up than K8s, and it might get you 90% of the way there. Whether or not that’s enough is gonna be up to you and your team to decide. (also I’m not a k8s pro so there’s probably other people better equipped to answer that)

Thanks @Vinnie! OK so it looks like it's not possible with ECS anyway to use different resources within a job - it does look possible with the k8s_job_executor by the looks of it though https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#per-job-or-per-op-kubernetes-configuration

It’s possible that you can also specify op specific memory and CPU on ECS and it’s just undocumented (I never had to, so I’m not sure). taking the liberty to cc @daniel who will probably know straight away

you can't specify op specific memory in ECS b/c we don't have a way to run each op in its own ECS task yet, but there's an issue tracking it here and I'm hoping we can work on it soon: https://github.com/dagster-io/dagster/issues/9671

Thanks @daniel - I'll keep an eye on the issue. In the meantime. Is there an example somewhere of achieving this with K8s ?

yeah, k8s example is here: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#per-job-or-per-op-kubernetes-configuration