Hi everyone. I'm investigating dagster and I'm str...
# deployment-ecs
t
Hi everyone. I'm investigating dagster and I'm struggling with how to use ECS with it. My use case is simple and I would prefer to deploy dagster locally, but there are a couple of steps in my data pipeline which require running on ECS. From what I can tell the ECSRunLauncher expects dagster to also be deployed on ecs to get configuration options from the context. Is there a way to configure all the necessary options outside of this? Based on the deploy_ecs example - it looks like another option is to set up a gRPC server. Is that a feasible route to call from a local deployment? Bearing in mind in all of this, is that my infrastructure knowledge is limited and we don't have an infrastructure team, I guess using dagster cloud is an obvious option - but it feels like it would make a lot of the downstream tasks that don't rely on ECS more laborious to develop? Any advice greatly appreciated.
j
Re: developing on Dagster Cloud, check out our docs on both unit testing with Dagster and developing against Dagster Cloud branch deployments: • https://docs.dagster.io/concepts/testinghttps://docs.dagster.io/guides/dagster/branch_deployments It’s usually possible to write your code in such a way that it’s run launcher agnostic and you rarely have to interact with ECS during the development process. Re: deploying to ECS, the ECSRunLauncher no longer requires that the rest of the Dagster runs on ECS. You can configure the ECS task directly instead of letting Dagster try to infer defaults: https://docs.dagster.io/deployment/guides/aws#customizing-the-launched-runs-task
t
thanks that helps