https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alec Ryan

07/07/2022, 7:43 PM
Say I want to launch a backfill for a partitioned_config. Is there a way to extract the start/end date from backfill ? A specific case would be connecting to a 3rd party API where I can request data between two dates. I'd like to run a backfill but only request data once between the start and end date rather than launching multiple requests for each date
p

prha

07/07/2022, 8:47 PM
I’m not 100% sure that I’m following… are you asking how to get the partition range for an already created backfill, or how to create a new backfill for a given range of partitions?
a

Alec Ryan

07/07/2022, 9:25 PM
The former
Sorry if unclear
The goal is to get the date range of a backfill as an input to a function rather than run a function for every partition in a date range of a backfill
p

prha

07/07/2022, 9:35 PM
Right now, the backfill object contains a list of partition names… It would have to load the partitioned config to know if they represent a contiguous range
For example, I could have a dynamic partitioned config which emits
A
,
B
,
C
,
D
,
E
. And I could create a backfill over that whole range. But later on, I could change the partitioned config to be
A
,
B
,
C
,
C+
,
C++
,
D
,
E
. The backfill itself is immutable, but the range would change from
A-E
=>
A-C
+
D-E
.
In code, you fetch the backfill and load the partitioned config and derive the appropriate set of ranges and write your functions to operate over those ranges.
a

Alec Ryan

07/07/2022, 9:54 PM
Thanks @prha I'll give this a shot
6 Views