Hi everyone Just getting started with Dagster but loving it

Question

Hi everyone Just getting started with Dagster but loving it so far Impressive work I ve been trying to run a small distributed pipeline using multiprocessing but I m getting a complain about a missing RunLauncher This page <https docs dagster io docs learn guides partitioning backfill> states that you indeed need to make sure you have a RunLauncher set up But I m not sure how to set one up to run the backfill locally

alex · Answer

The solution for now is to make sure you have `dagit` running at an accessible location and use `RemoteDagitRunLauncher` example shown in here <https docs dagster io docs deploying instance>

SamSinayoko · Answer

amazing many thanks Alex

SamSinayoko · Answer

So there s no way for now to run distributed backfills without having dagit running I d like to introduce Dagster to my team We have a lot of ETL jobs that use command line scripts with multiprocessing so that d be a good place to start It d be great to be able to do this without dagit though Happy to help implement this if you point me in the right direction

alex · Answer

gt no way for now to run distributed backfills without having dagit running You can write your own `RunLauncher` implementation to hand off the pipeline execution to some long lived process that can take care of them

alex · Answer

were working on one that will be available in `0 8 0` in a few weeks

alex · Answer

if you dont care about orphan processes and you have a single point of truth for loading the repo you can write a simple `RunLauncher` that just does fire and forget with `subprocess Popen`

SamSinayoko · Answer

will give that a go thanks