Say I want to launch a backfill for a partitioned config Is dagster #ask-community

Say I want to launch a backfill for a partitioned_...

Alec Ryan

07/07/2022, 7:43 PM

Say I want to launch a backfill for a partitioned_config. Is there a way to extract the start/end date from backfill ? A specific case would be connecting to a 3rd party API where I can request data between two dates. I'd like to run a backfill but only request data once between the start and end date rather than launching multiple requests for each date

prha

07/07/2022, 8:47 PM

I’m not 100% sure that I’m following… are you asking how to get the partition range for an already created backfill, or how to create a new backfill for a given range of partitions?

Alec Ryan

07/07/2022, 9:25 PM

The former

Alec Ryan

07/07/2022, 9:26 PM

Sorry if unclear

Alec Ryan

07/07/2022, 9:26 PM

The goal is to get the date range of a backfill as an input to a function rather than run a function for every partition in a date range of a backfill

prha

07/07/2022, 9:35 PM

Right now, the backfill object contains a list of partition names… It would have to load the partitioned config to know if they represent a contiguous range

prha

07/07/2022, 9:38 PM

For example, I could have a dynamic partitioned config which emits

. And I could create a backfill over that whole range. But later on, I could change the partitioned config to be

C+

C++

. The backfill itself is immutable, but the range would change from

A-E

A-C

D-E

prha

07/07/2022, 9:39 PM

In code, you fetch the backfill and load the partitioned config and derive the appropriate set of ranges and write your functions to operate over those ranges.

Alec Ryan

07/07/2022, 9:54 PM

Thanks @prha I'll give this a shot

6 Views

Open in Slack

Previous Next