Hey, I’m starting to setup Dagster in ECS for our team so we can hopefully migrate from airflow, but I’m unsure on how to proceed in fetching updates from different repositories, here’s what I roughly planned: ECS Cluster running a service with dagit, daemon, and a gRPC for each part of our user code as a standalone repository (one for our internal ETL tool, one for the transformation part, etc). With the base scripts for them, as well as schedules/jobs/etc defined within each of these repos. Now, as far as I can tell this would require us to restart the service so the latest version of the scripts is fetched every time whichever one of these components is fetched. This seems to me like a bit of an overkill and I’m assuming I’m overlooking a different way to do it a little more cleanly, but couldn’t find any exact information as I scoured the web. Any ideas of how best to do it/how your teams implemented it? This exact problem was a non-issue when running Airflow as we’re outsourcing all the computing to separate ECS tasks/lambdas, so we can deploy the code independently of airflow and it would just execute the lambda on schedule.

Hi Vinnie - the only piece of the deployment that you need to restart when your code changes is the gRPC service for the code that changed - Dagit and the daemon are able to reload your code automatically when those changes happen. Does that help at all? Or is the fact that you need to restart anything when code changes what feels off?

That’s more or less what I had understood too, I guess my issue with it is because I planned to run everything in the same ECS service (so everything would restart every time). The “better” solution would be to just split it into a main service running dagit/daemon and then one service for each repo. Did I get that right?

That's right, yeah - dagit and the daemon could be separate services too, since they might have different autoscaling behaviors / resource requirements We have an example that has that setup here: https://github.com/dagster-io/dagster/tree/0.14.12/examples/deploy_ecs

Got it. I couldn’t spin it up with the docker compose since the company has incredibly strict guardrails, so I’m trying to piece it together with terraform. The info is helpful though, thanks a ton!