Hi all! I have been running multiple instances of...
# dagster-serverless
Hi all! I have been running multiple instances of dagster for a few years now. I've created docker compose files which are used to build docker images and then spin up docker containers. I have a container per dagster "component" (daemon, webserver, postgres, repo), all running on the same VM (which is always up). It's time to start thinking about being able to scale up the workload ability, i.e. I'd like to be able to distributed runs of multiple concurrent jobs. It would be ideal to be able to serverlessly spin up a new container for each job. Not all jobs are equal though - some jobs might require minimal compute, but some might need GPU. The same docker image could be used for every job. I have seen https://docs.dagster.io/deployment/guides/aws#launching-runs-in-ecs, which is along the lines of what I'm thinking. We use GCP where we can, but will need to remain cloud agnostic, or at least be able to support the big three. If any of this resonates with anyone or if you have any pointers, please let me know.