I like to run dagster on my local workstation before I push to AWS, but this ECS run launcher obviously can’t be run locally in a docker-compose configuration. Is there a way to change the run launcher based on the mode or environment rather than having it hard-coded into the dagster.yml file?

I think you can make do with custom

dagster.yaml.sh

where you set

RUN_MODE

env var and populate config there. basically you use RUN_MODE to generate dagster config (we do it w our setup)

interesting. I’ll give it a shot

Yeah - there’s no built-in way to flip run workers. It all has to be sourced from the yaml. But like Max suggests, there are probably creative solutions to interpolate your yaml based on your environment.

yeah my first shot was to create a duplicate dagster.yml and use docker-compose volumes section to overwrite the “production” yaml with the “stage”. That worked. But I don’t like the idea of maintaining two separate yaml files. So I’m gonna give Max’s suggestion a go

Here’s an option: LocalStack. Unfortunately you’d have to get the Pro (paid) version to get ECS. Haven’t tried it yet myself, but it looks like an interesting option for testing code locally that relies on AWS services.

I haven’t personally used the localstack implementation of ECS and I’m definitely curious how well it mocks the service. Internally, we ended up writing a lot of boto stubs to do local development of the EcsRunLauncher. But we didn’t mock it to the point where you could actually use it end-to-end. Some localstack implementations are quite good though - I’ve had a lot of success using it to run things like S3 and SQS locally (I think they just use moto under the hood for both).