Sam Rausser05/30/2020, 7:05 PM
in another thread or process and loop? like say a kafka consumer spilling to disk after N bytes and then running
on a pipeline that finds the file and processes it etc.
alex06/01/2020, 1:33 PM
implemented to check for the data to be processed. You can see a conceptually similar approach used for backfilling here https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/schedules.py
spun up execute_pipeline in another thread or process and loopthe only real issue i expect you to face here is managing the life cycle of the process/thread and making sure it doesn’t negatively impact the outer pipeline execution. Things will change a bit in a few weeks when we release
and there could be another option