Has any work been done to date to support executions via kna

Question

Has any work been done to date to support executions via knative and therefore google cloud functions

mrdavidlaing · Answer

Depends what you mean by extensions It is possible to run dagster on Cloud Run in single container mode

mrdavidlaing · Answer

I deploy a simple web runner flask app that runs `execute pipeline ` when triggered by a HTTP request from Cloud Scheduler I run `dagit` locally connecting to the same GCP Postgres instance to observe the run output

mrdavidlaing · Answer

No fancy spread load across multiple containers but it can work out very inexpensive since you re only paying for the time your pipelines are actually running my simple pipeline that runs once day costs ~ 0 01 for 385 47 vCPU second day

Agon Shabi · Answer

Nice that s very similar to how I ve been using it I have a persistent SQL Server instance on demand ephemeral compute on prem triggerable via HTTP request and a way to run dagit pointed at the database I might try and license a managed k8s service but since my workloads don t currently require any special hardware resources I thought it might be simpler to forego the k8s process overhead and rely entirely on cloud functions for execution of solids and maybe app engine for dagit

mrdavidlaing · Answer

One problem I haven t been able to solve with hosting Dagit on GCP Cloud run is HTTP authentication If you come up with a solution that isn t too expensive I ve love to learn what you did