https://dagster.io/ logo
p

Philippe Laflamme

01/11/2023, 7:50 PM
👋 Hello! I’m completely new to dagster and have mostly only read documentation. So I’m sorry if this is mentioned anywhere that I missed. I have a question regarding backfilling. Say I have a daily partitioned asset defined that requests the data using a http request and produces a file to disk, say using parquet. I’d like to backfill this for a whole year. From my understanding, that would make one HTTP request per partition. But the API I’m using supports larger batches, say per month. How would I go about backfilling a daily partitioned asset, for a full year, using monthly requests (12 HTTP calls for 365 files)?
s

sandy

01/12/2023, 12:17 AM
Hey @Philippe Laflamme- we don't currently have the ability to break up a backfill into multiple runs that each operate on a range, but you can launch a single run that operates on a range. I just wrote up a short guide on how to do this: https://github.com/dagster-io/dagster/discussions/11653. Here's an issue where we're tracking more general requests on this topic: https://github.com/dagster-io/dagster/issues/8706
p

Philippe Laflamme

01/12/2023, 1:32 AM
Awesome, thanks!