Hi all, does anyone know if Dagster supports executing Google Cloud Dataflow jobs, specifically those that are implemented in Scala? We have a substantial amount of code in that world and are interested in potentially using dagster as an orchestrator for those jobs.

👋 hey Andrew, we haven’t directly built support for Cloud Dataflow, but it should be pretty straightforward to implement a Dataflow resource if you’re interested in giving it a shot

@nate sure, any pointers on where to start with such a resource?

take a look at the existing GCP integrations https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-gcp/dagster_gcp

yeah, here’s a simple one: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/resources.py here’s an example of a more involved resource definition: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/redshift/resources.py

excellent, thanks

sure, when you say docker containers do you mean on GKE?

gke

awesome. and are you interested in instigating execution of externally-defined containers, or running Dagster solids as containers? for the former, its pretty straightforward to write a solid where the function body kicks off execution of externally-defined containers. for the latter, we’ve got a Helm chart which you can use to deploy Dagster to GKE (we use GKE for our Dagster deployments internally) and execute pipelines there

it’d be the former

yeah, I don’t know if we have a great example of that in the codebase, but we use this for instigating execution solids-as-containers and entire pipelines-as-containers on k8s. As long as you’re using the Python kubernetes API and have the appropriate k8s privileges in place, it should be pretty straightforward to instigate container execution on k8s from a solid

oh got it