Hi everyone. Just getting started with Dagster but...
# announcements
s
Hi everyone. Just getting started with Dagster but loving it so far. Impressive work! I've been trying to run a small distributed pipeline using multiprocessing, but I'm getting a complain about a missing RunLauncher. This page https://docs.dagster.io/docs/learn/guides/partitioning/backfill states that you indeed need to make sure you have a RunLauncher set up. But I'm not sure how to set one up to run the backfill locally.
a
The solution for now is to make sure you have
dagit
running at an accessible location and use
RemoteDagitRunLauncher
- example shown in here https://docs.dagster.io/docs/deploying/instance
s
amazing, many thanks Alex!
So there's no way for now to run distributed backfills without having dagit running? I'd like to introduce Dagster to my team. We have a lot of ETL jobs that use command line scripts with multiprocessing, so that'd be a good place to start. It'd be great to be able to do this without dagit though. Happy to help implement this if you point me in the right direction.
a
no way for now to run distributed backfills without having dagit running?
You can write your own
RunLauncher
implementation to hand off the pipeline execution to some long lived process that can take care of them
were working on one that will be available in
0.8.0
in a few weeks
if you dont care about orphan processes and you have a single point of truth for loading the repo you can write a simple
RunLauncher
that just does fire and forget with
subprocess.Popen
s
will give that a go, thanks!!