Step launchers and expanded support for PySpark on...
# announcements
m
Step launchers and expanded support for PySpark on EMR and Databricks We've introduced a new
StepLauncher
abstraction that uses the resource system to allow individual execution steps to be run in separate processes (and thus on separate execution substrates). This has made extensive improvements to our PySpark support possible, including the option to execute individual PySpark steps on EMR using the
EmrPySparkStepLauncher
and on Databricks using the
DatabricksPySparkStepLauncher
The
emr_pyspark
example demonstrates how to use a step launcher.
s
Thank you @Ben Sully for this contribution!
👍 1
🎈 2