Setups for different environments? I'm trying to keep 1

Dockerfile

for both DEV and PROD, but I need to have different

dagster.yml

files for each (local machine vs AWS ECS). The

workspace.yaml

changes too (host name in ECS is

localhost

while its the container name in docker-compose locally). Also unsure how to just pull in the repos that I've setup separately for development/testing and production? Is there a recommended way / best practice for this?

I’ll preface this by saying that this is based on my understanding of the system and is possibly (maybe even likely?) wrong, but I’m sure someone from Elementl will read and can correct whatever wrong assumptions I’m passing. 1. Your

dagster.yaml

should be in your

$DAGSTER_HOME

, you can have different ones in local and cloud environments and the local one doesn’t need to point to a postgres instance, doing so would mean it accesses the same event logs, schedule storage and run storage, which you probably don’t want (my local

dagster.yaml

file lives in

~/dagster_home

and only sets the log level as

DEBUG

, for example). I don’t develop locally with docker-compose though, usually spin up a dagit instance from inside the user code location/repository I’m currently working on, but you should be able to map your docker-compose to a different

dagster.yaml

2. I don’t completely understand what you’re doing with the

localhost

, but you can always have multiple files and pass the command, as you mentioned. 3. Running

dagster-daemon run --help

on my terminal does show the

-w

flag, might be missing in the docs (didn’t check), but it does work. 4. Check out the fully featured project. It shows a pattern for dynamically setting the repository depending on the deployment. I think there’s also a way to specify the exact repo the workspace fetches if you have multiple decorated functions, but I can’t seem to find an example for that (and it’s not a pattern I’ve used)

hey, yeah thanks for the detailed reply. I should probably just run

dagit

locally but i like the local deployment to match the production as closely as possible so i'm doing multi container deploys locally too. My issue with the

daster.yaml

file is that (a) you can't specify it in the command so you have to do it in the Dockerfile which means you can't reuse the same file for local and production (I have a hack that does it, i'l post it bellow). (b) i need to have 2 files as production uses the

dagster_aws.ecs

run launcher and local uses

dagster_docker

plus s3 storage for

compute_logs

etc. I've been working from the

examples/deploy_ecs

example. Finally got it going but its awkward (see Dockerfile bellow):