Is there a recommended way to speed up the submiss...
# dagster-feedback
Is there a recommended way to speed up the submission of large backfills to the
? Here’s some context: • Self hosting Dagster and on 1.3.11. Using the
+ Multiprocess Executor with the
• We’re submitting multiple large backfills (in the 1000s of runs) and have noticed it takes quite awhile for jobs to be enqueued. It appears that backfills use the same Queue, and if you submit sequential backfills all jobs in earlier backfills must be enqueued before later backfills can be enqueued. This is somewhat problematic if we want to start different backfills in parallel. • Here’s our config for `QueuedRunCoordinator`:
Copy code
class: QueuedRunCoordinator
    dequeue_interval_seconds: 5
    dequeue_num_workers: 4
    dequeue_use_threads: true
    max_concurrent_runs: 100
We aren’t really ever hitting
; the issue is more about enqueue speed. Should we look into implementing our own custom
Also fixing broken hyperlink in these docs 🎅
Hey Alex - this is an old post but since it came up briefly in the doc you sent us: What I think we would need to resolve this is to enqueue runs in parallel in the backfill daemon (akin to the solution here that we used to speed up sensor ticks that submit many runs in parallel:
very reasonable feature request if you wouldn't mind filing an issue
🎅 will do!