Dagster External Execution Protocol (ext) Hey everyone. Just want to highlight a github discussion we just posted, detailing a feature coming out in our next release (currently scheduled on September 28th). We call it the Dagster External Execution Protocol or “ext.” ext has a few goals: • Make it easier to orchestrate existing code with Dagster with little to no modifications to that existing code • Make it easier for centralized data and data engineers to onboard stakeholder teams onto Dagster incrementally • Run code in external environments and easily get streaming logging and structured metadata back into Dagster’s UI and tools • Formal support for separation of orchestration and business logic environments • First-class multi-language support in Dagster Please read the github discussion for more details and to give feedback, but how are achieving these goals is shipping a toolkit for greatly improved integrations with external environments such as Kubernetes, Spark, and other external runtimes. It standardizes parameter/context passing, as well as streaming logs and metadata back to Dagster. It is also extensible and pluggable, so that users can build on it to orchestrate their own customized environments. We’ve created a channel #dagster-ext to follow updates and discuss in more detail. Thanks all! Looking forward to getting this out to everyone.

As someone who works on a team that extensively leverages external distributed compute, also with lots of pipelines in Scala, I can't wait to get my hands on this!

It would be great to have a Live demo of this feature. I read it through, but I am just sparingly understanding it.

@Binoy Shah great idea we will throw that together