https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
d

Dusty Shapiro

02/06/2023, 9:22 PM
I would be curious how other people here have Dagster setup for both local and deployed env? Our deployed environments (via Helm), are in a great state, but I’ve recently turned my attention to giving the local development experience a bit of love.
m

Mark Fickett

02/06/2023, 9:26 PM
We run our pipeline locally as well (development + previous generation deployment). We use the multiprocessing + forkserver backend.
s

Stephen Bailey

02/07/2023, 2:20 AM
some of the quality of life improvements released recently have made our local experience great. moving to
.env
files for all secrets means dagit reads it on startup, and the new
dagster dev
commands takes care of spinning up sensor daemons for us. we used to have a docker compose flow, and now we can just use our poetry shells and regular aws credentials
a

Alexandre Guitton

02/07/2023, 9:21 AM
Still in POC Dagster, but for local development we have defined two stages: • Pure implementation phase: with the
default runLauncher
and
.env
file. • Once implementation is finished: we switch to the
K8sRunLauncher
which uses the deployed config in staging. This way, we still develop locally, but we run jobs on our target cluster: this allows validation of network accesses, IAM policies and environment configs (Secrets and ConfigMaps) All this using
docker-compose
(with a unique docker image from dev to prod). If you want an example you can take a look here
6 Views