https://dagster.io/ logo
#deployment-ecs
Title
# deployment-ecs
r

Roberto Tolosa

03/17/2023, 4:12 AM
hey all – we've trying to deploy dagster on ECS, but we've struggled to find a recent enough template to go off of. the guide from the dagster repo (link) says "_Once you're done experimenting with this reference deployment, you should destroy the AWS resources it creates._", so i can't whether this is meant to be used in production. are there any good recent deployment templates that you'd recommend following?
m

Mike Atlas

03/17/2023, 2:24 PM
My team is using a slightly customized deployment we built using this reference in production. I think that the "_should destroy"_ language there is not really accurate
thank you box 1
j

johann

03/17/2023, 4:39 PM
That docker compose example is our current recommendation.
Once you’re done experimenting with this reference deployment, you should destroy the AWS resources it creates.
is just to try to avoid someone getting charges by accidentally leaving a demo deployment up 🙂
👍 1
r

Roberto Tolosa

03/17/2023, 5:31 PM
@johann so does that mean that this is or isn't meant to be used in production? i couldn't quite tell now haha
sounds like it definitely usable in production, but that text is just reminding people to tear it down if they were only temporarily using a given deployment as a demo?
☝️ 1
👍 1
j

johann

03/17/2023, 5:56 PM
Yes you can use it in production
r

Roberto Tolosa

03/17/2023, 8:10 PM
love it, thanks!
and thanks for all the many questions you've answered for me, johann! excited to have a solid working setup soon blob clap
a

Arnoud van Dommelen

03/20/2023, 10:41 AM
Hi Roberto, we are also (partly) using docker-compose solution in production. However, do consider changing to CloudFormation when the infrastructure is fully built, as this gives you more flexibility in terms of configuration! :)
m

martin o leary

03/24/2023, 1:20 PM
Hey @johann - sorry to piggy back on this thread but I am trying to deploy dagster on AWS and looking at the example I am sort of confused but my understanding of devops is basic at best. Should all this code live in the same repo as the user code? Our setup is; develop in a devcontainer locally, push changes to GitHub where we have an actions workflow that builds our container (user code) and stores in GHCR Would the ECS example code be in a separate repo or the same as the user code? I'm not sure how to adapt the example basically.
m

Mike Atlas

03/24/2023, 1:54 PM
fwiw my team has two separate git repos (we are self-hosting dagster) - one git repo for dagster services (daemon and dagit) that has IaS (terraform), docker compose, dagster config, etc. Then we have another git repo for our workspace/pipeline service code (actual jobs/graphs/etc, also terraform)
The latter is where all the business logic lives and gets many deployments via CI/CD regularly. The dagster git repo we only upgrade and deploy roughly 1x or 2x a quarter
In total there are 3 services in our ECS environment: dagster daemon, dagster dagit, and our (ETL / biz logic) pipeline code service
m

martin o leary

03/24/2023, 2:11 PM
Thanks Mike - so your business logic repo, that builds a container and tags it something like <biz logic name>:latest and that's referenced in your services repo at this point?
m

Mike Atlas

03/24/2023, 3:24 PM
@martin o leary yes, well, sort of; we aren't including dagit... let me share a gist of the rough Dockerfile
m

martin o leary

03/24/2023, 3:30 PM
Thanks for putting that together. i'm assuming that your user code (all your jobs / assets) is under the
pipeline
dir in your example?
👌 1
👍 1
OK I'm very confused 😂 I think I need to go deploy some stuff on ECS outside of dagster because that is new also
m

Mike Atlas

03/24/2023, 3:41 PM
Ah, yeah, at least this Dockerfile would be fine for k8s or ECS
🙏 1
We are using the ECS service discovery (includes automatic private route53 entry creation) feature so that the dagster config can resolve the user code service
That's part of the docker-compose example, but it's opaque since it triggers a cloudformation job to establish the discovery routes. On my team we redid this as terraform instead...
👍 1
7 Views