For spark pipelines, is there a plan for dataproc serverless support?
We're planning on migrating to gcp and since our workloads sporadic (only a few pipelines with intensive resources requirement, run twice monthly), using an always on dataproc cluster might not be ideal
07/08/2022, 7:40 PM
It’s not in our immediate roadmap, but can file a GH issue to track…
@Dagster Bot issue add support for dataproc serverless