Step launchers and expanded support for PySpark on EMR and Databricks We've introduced a new

StepLauncher

abstraction that uses the resource system to allow individual execution steps to be run in separate processes (and thus on separate execution substrates). This has made extensive improvements to our PySpark support possible, including the option to execute individual PySpark steps on EMR using the

EmrPySparkStepLauncher

and on Databricks using the

DatabricksPySparkStepLauncher

The

emr_pyspark

example demonstrates how to use a step launcher.

Thank you @Ben Sully for this contribution!