https://dagster.io/ logo
#random
Title
g

George Pearse

03/17/2022, 11:08 AM
How do people balance the engineering benefits of backfills (testability / idempotence) against the runtime benefits of having a different flow for an initial load? In my case a huge psycopg2 copy_from to insert into Postgres and MongoDB collection call without any filter (executed in large batches). Or are backfills more often just used when a job's been failing for a few days?