In case it helps, this is how I can handle the above in Luigi. 1. In Luigi, a task definition includes a list of dependencies, and Luigi automatically runs independent tasks concurrently. 2. In Luigi, I could remove output from Task A and C, and rerun Luigi. That way, Luigi knows not to rerun B.