https://dagster.io/ logo
#ask-community
Title
# ask-community
s

shigeaki onoda

07/20/2023, 12:08 PM
Hi. I'm new to dagster. I’m loving it😍 I launched dagster components on ECS. It works well. My question here is we always include dagster projects along with same repository that docker compose resides? This is because we adapt super duper strict policies on github repo access even for colleagues. I’m finding sorta apply command in k8s cronjob, which can be called anywhere as long as you specify a cluster and namespace.
j

johann

07/20/2023, 10:02 PM
My question here is we always include dagster projects along with same repository that docker compose resides?
Not sure if I’m understanding correctly, but if you’re deploying with docker compose you’d just need the docker images produced by the docker code in the gh repository
s

shigeaki onoda

07/21/2023, 3:12 AM
via rpc service? What I meant is similar as this post: https://dagster.slack.com/archives/C014UDS8LAV/p1689594909074219 My concern is we eventually distribute same config files such as workspace.yaml and dagster.yaml across all repositories with the way in this example. https://github.com/dagster-io/dagster/tree/1.4.0/examples/deploy_ecs
j

johann

07/21/2023, 3:57 AM
You could set up each repository to deploy its own grpc server. They would each need a dagster.yaml config (could load this in to ECS tasks, doesn’t need to be in the git repo) and only the dagit service needs a workspace.yaml, that would only need to be updated when you add/remove a code location or change its address
s

shigeaki onoda

07/21/2023, 5:29 AM
So the only coupling variable is my code location in workspace.yaml? Here is what I think off the top of my head. Dagster core component repo • dagster.yaml • workspace.yaml that need to include each repo code location • Whatever ECS definition spin up dagster daemon and dagit My repo has my actual workflow code • mycode • dagster.yaml that is separated one in the core repo. • Dockerfile • Ecs task definition ( whatever IaC like CFn or terraform) • No workspace.yaml?
j

johann

07/21/2023, 5:32 AM
That sounds right to me
Ideally the dagster.yaml for the code locations matches the one for dagit/daemon. It doesn’t have to but there’s some potential gotchas (e.g. it certainly needs to point at the same db). Secrets in the dagster.yaml can be loaded from env vars.
s

shigeaki onoda

07/21/2023, 5:37 AM
For sure. No one want to define same things across multiple locations.
j

johann

07/21/2023, 5:37 AM
I’m not an ecs expert, do you know if there’s a way to inject a file (dagster.yaml) in to ecs containers via the task def? Akin to kubernetes volume mounts? That’d be convenient here. From aws parameter store and secrets manager I only see injecting env vars
s

shigeaki onoda

07/21/2023, 5:39 AM
I’m not also an ECS expert but we might try EFS volume to share same files across many tasks
👍 1
2 Views