Hi Dagster folks! We’re evaluating orchestration platforms right now. While many of us use Python, some of our users need to code operations in C++ and would like to build them within a Docker container. Is there a good way to do this in the Dagster ecosystem?

Hey MIchael - do you know yet where you would be running your Dagster agent? (K8s/ECS/serverless/etc.)

K8s

That's what I was hoping you'd say - we have a

k8s_job_op

that lets you run an arbitrary Docker container as a Dagster op: https://docs.dagster.io/_apidocs/libraries/dagster-k8s#ops

Thanks, we’ll take a look!

Out of curiosity, what’s the best way to go about doing this in dagster serverless? We don’t have anything right now, but are scoping out something that may land here ~Q2.

We don't have support in Serverless for launching arbitrary containers/tasks right now unfortunately - eventually we may, but in the short term we would probably steer a user towards hybrid deployment for that use case

I'm on the same boat! We're mostly python but have some workloads in Docker. Would it be possible to run things like https://cloud.google.com/batch/ jobs from Serverless?

You could absolutely kick off a job like that from within a dagster op, yeah - as long as they have an api that can be invoked from Python