Can Dagster be run on GCP Cloud Run? I see a guide for AWS ECS and one for GCP GCE but not for Cloud Run. In particular I’m wondering about the scale-to-zero feature of Cloud Run/Knative which is not the case with ECS.

Hi Ian - we haven't yet written a launcher for Cloud Run, but I can't think of any reason why it wouldn't be possible - the system is pluggable at that layer to support custom run launchers

I take it that would be if you’re running Dagster on a Kubernetes cluster and then configure separate Run Launchers as Cloud Run containers? But it wouldn’t work if Dagster itself was stored on a single Cloud Run container?

I was imagining configuring a run launcher on the instance that creates a new cloud run container for each run, yeah. I don't know enough about cloud run to speak to whether it would make sense to host the long-running dagster services described here on cloud run https://docs.dagster.io/deployment/overview

Cloud Run does now provide “always-on” CPU allocation and you can set “minimum instances” for the container itself. I’m wondering if this would be enough to run Dagster and potentially longer running services? https://cloud.google.com/blog/topics/developers-practitioners/use-cloud-run-always-cpu-allocation-background-work

That sounds promising to me, yes!

How technically involved is it to create a new Run Launcher?

it's not completely trivial, but also not huge - take a look at the DockerRunLauncher for an example: https://github.com/dagster-io/dagster/blob/e1adadd33cecf01474ec1c232b5b52b68ab1210[…]/libraries/dagster-docker/dagster_docker/docker_run_launcher.py Ultimately it involves launching a task in the service of choice using a specific command that is pulled from the run object that's passed in

Thanks @daniel I’ll take a look through things. If you know of any other examples then let me know. They may be helpful.

I wonder if this thread went anywhere. This seems like a really interesting low-lift setup. I can imagine using cloud run instances for the daemon, digit, then a jobs instance for the executor workloads?

FWIW, we would find this really valuable as well. Prioritizing work on this is perhaps the roadblock for us to try out Dagster for real..