Hi team, I had a request some time ago to support multiple partitions for a single job so that we don't have to create one job for day. I was told that asset partition would be able to handle this. Is it possible to do this with Assets now?

cc @sandy

Hey @Arun Kumar - alas, we're still not there yet, but most of the pieces are in place. Here's the issue where we're tracking this: https://github.com/dagster-io/dagster/issues/8706. Does what's described in there line up with what you're looking for?

Thanks @sandy. Yes, that mostly matches with what I am looking. Not sure if this is already covered in that issue. Would also be great if we have some way to specify a custom setting from the UI while running backfill to determine the correct set of the ranges that need to be handled in each step. i.e., From the backfill UI, in addition to selection of date ranges, some way to specify Weekly or Monthly range so that the backing partition function can return appropriate date ranges (week or month ranges) based on the custom setting.

Got it - how would you decide whether to do weekly vs. monthly?

I usually determine based on the scale of the data that I am backfilling. For most of the sources, monthly ranges work, but there are some high scale source where I would do a weekly backfill to avoid timeouts.

I see - would it make sense to set a policy on the on the asset and/or job, or would you need to be able to make the decision when launching the job?

We actually dynamically generate the jobs and partitions based on our internal config system. I usually make the decision when launching the job and it would be great to have that flexibility

Got it - that make yes sense. Thanks for the input!

@sandy saw this on the previous release changelog. Just curious if this is something related to what we are talking about

yeah - that's one of the building blocks

I see, but is that not enough to make a job backfill multiple partitions? I was just curious to know if I can build some hack around this right now, before the feature is available for use

that PR makes it easy to find out all the partition keys that are between a start and an end date, but we don't yet have the functionality to launch a run that targets all those partitions and use them to make things show up in the UI