I would like to better understand the the execution model of dagster, RunLauncher, ExecutionPlan, Executor, etc. Where should I start?

the link in the docs is busted but heres a picture from the repo that was a snapshot of how things were layered as of March 2020

Thanks, I will try to have a look also at the libraries

RunStep?

Sorry I meant

ExecuteStep

https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/execution/plan/execute_step.py

pyspark

uses the

StepLauncher

if thats what you mean

Maybe it's just my missunderstanding of it, but it seems that you can create runners at different levels, like step or plan?

ya I think part of what you are referring to is the recently added

StepLauncher

https://dagster.phacility.com/D2688

What should I implement if I want to create a custom runner? For example I want to write something that will take a whole compiled pipeline (is this what is called an ExecutionPlan?) and then execute it using its own internal mechanism.

likely an

Engine

so the examples to reference would be the celery and dask engines

How is that different from the k8s one?

they are all similar in nature - they are effectively handling how to execute each individual step, usually the key aspect being federating out the work somewhere and managing that

So, is it fair to assume that every Launcher will execute something out-process?

so our

k8s

deployment example uses a

celery_k8s_job_executor

which is submits tasks to a celery queue for each step that will in turn submit k8s jobs

Ok, I think I understand better... but at the same time the k8s deployment launches a dagster job to orchestrate the pipeline execution or is it done from the original process?

so the

RunLauncher

determines where the

Executor

( I called it Engine above by mistake) or run master is operating then the

Executor

decides how to handle each step and a

StepLauncher

is a way to special case steps from the default

Executor

behavior - the current version does this to ship

pyspark

solids to a spark cluster

I think I get, that's a very helpful starter! 👍

no problem - good luck!

it seems like this diagram link is 404ing now -- did that image move?

danke