Can Dagster be run on GCP Cloud Run? I see a guide...
# ask-community
i
Can Dagster be run on GCP Cloud Run? I see a guide for AWS ECS and one for GCP GCE but not for Cloud Run. In particular I’m wondering about the scale-to-zero feature of Cloud Run/Knative which is not the case with ECS.
d
Hi Ian - we haven't yet written a launcher for Cloud Run, but I can't think of any reason why it wouldn't be possible - the system is pluggable at that layer to support custom run launchers
i
I take it that would be if you’re running Dagster on a Kubernetes cluster and then configure separate Run Launchers as Cloud Run containers? But it wouldn’t work if Dagster itself was stored on a single Cloud Run container?
d
I was imagining configuring a run launcher on the instance that creates a new cloud run container for each run, yeah. I don't know enough about cloud run to speak to whether it would make sense to host the long-running dagster services described here on cloud run https://docs.dagster.io/deployment/overview
i
Cloud Run does now provide “always-on” CPU allocation and you can set “minimum instances” for the container itself. I’m wondering if this would be enough to run Dagster and potentially longer running services? https://cloud.google.com/blog/topics/developers-practitioners/use-cloud-run-always-cpu-allocation-background-work
Actually Cloud Run now offers “Cloud Run Jobs”. Would this be suitable for Dagster in the same way that the ECSRunLauncher launches an ECS task per run?
d
That sounds promising to me, yes!
i
How technically involved is it to create a new Run Launcher?
d
it's not completely trivial, but also not huge - take a look at the DockerRunLauncher for an example: https://github.com/dagster-io/dagster/blob/e1adadd33cecf01474ec1c232b5b52b68ab1210[…]/libraries/dagster-docker/dagster_docker/docker_run_launcher.py Ultimately it involves launching a task in the service of choice using a specific command that is pulled from the run object that's passed in
We've absolutely had commnunity members create run launchers in the past
i
Thanks @daniel I’ll take a look through things. If you know of any other examples then let me know. They may be helpful.
d
I wonder if this thread went anywhere. This seems like a really interesting low-lift setup. I can imagine using cloud run instances for the daemon, digit, then a jobs instance for the executor workloads?
v
FWIW, we would find this really valuable as well. Prioritizing work on this is perhaps the roadblock for us to try out Dagster for real..
534 Views