The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hi :wave: I am looking into the <https://docs.dagster.io/integrations/spark|Spark integration documentation> so that I can launch spark jobs on AWS EMR. Do I understand correctly that there is no OP which will launch an EMR  cluster and then run a step on it, and that instead the EMR cluster should already be running and Dagster can add / launch a step on that cluster? I am looking for something similar to Airflow's <https://airflow.apache.org/docs/apache-airflow/1.10.13/_api/airflow/contrib/operators/emr_create_job_flow_operator/index.html|EmrCreatejobFlowOperator>.

I guess I am answering my own question here but looking at the code <https://docs.dagster.io/_modules/dagster_aws/emr/emr|here>, we can call EmrJobRunner().run_job_flow(). Is this the correct way to use this class and method? :slightly_smiling_face:

Hey Daniel - good code-spelunking. Yes, that is correct!