Hi there, I have a couple (quite dense) questions to do with project structure and docker deployment on ECS. I have two projects I want to deploy in their own container (from this idea
but extending docker_example_pipelines service in docker-compose to maybe docker_project_A, docker_project_B). The logic here would be to have each project (which would be its own package like here
) in its own running container as a gRPC server:
workspace-projectA.yaml # For local development purposes
workspace-projectB.yaml # For local development purposes
dagster.yaml # This will be copied over in the docker images
workspace.yaml # This will be copied over in the docker images
I suppose my argument here is if we containerise both project_A (Dockerfile_projectA), project_B (Dockerfile_projectB) separately (in ECS say) if someone does a blunder and pushes broken code in project_B my project_A repository with all its pipeline code will still run and not fall apart? Is this a more standard approach to a multiple project based workflow for dagster? I watched a video of your talk
(33:40) about reloading repository but I felt the example was too simple having two separate repository modules with all logic within them, doesn’t really extend to larger projects so well.
Question 2, extending the above, if we make a change to project_A and add another pipeline say I would have to rebuild the image and push that. I’ve read about some git-sync options with Kubernetes (but not really an option with Kubernetes for us), would an alternative method be to set up a volume like EFS
to both containers holding project_A, project_B (read only) and then just have some CI/CD setup to push to EFS for every change? I’m hoping this might solve the problem of having to rebuild the image every time there is a change?
Sorry for the longwinded questions. Would really appreciate the feedback. Thanks.