I would be curious how other people here have Dagster setup for both local and deployed env? Our deployed environments (via Helm), are in a great state, but I’ve recently turned my attention to giving the local development experience a bit of love.

We run our pipeline locally as well (development + previous generation deployment). We use the multiprocessing + forkserver backend.

some of the quality of life improvements released recently have made our local experience great. moving to

.env

files for all secrets means dagit reads it on startup, and the new

dagster dev

commands takes care of spinning up sensor daemons for us. we used to have a docker compose flow, and now we can just use our poetry shells and regular aws credentials

Still in POC Dagster, but for local development we have defined two stages: • Pure implementation phase: with the

default runLauncher

and

.env

file. • Once implementation is finished: we switch to the

K8sRunLauncher

which uses the deployed config in staging. This way, we still develop locally, but we run jobs on our target cluster: this allows validation of network accesses, IAM policies and environment configs (Secrets and ConfigMaps) All this using

docker-compose

(with a unique docker image from dev to prod). If you want an example you can take a look here