Hi! Is it possible to have an asset partitioned da...
# ask-community
m
Hi! Is it possible to have an asset partitioned daily, but only run the schedule for the job associated with the asset on weekdays? The idea is the asset logic will handle data from the weekend (which should still show up as daily partitions) on Mondays. I saw this discussion on creating partitions only for business days, but does that mean weekends won't show up as partitions?
šŸ¤– 1
The other option is to leave the partition definition as daily, but only run the schedule on weekdays (with Monday covering the weekend), but if the Monday partition is re-materialized, then would it only clear out the data from Sunday and not include Saturday?
b
I think what you really want here is a WeekdayPartition, where the Monday partition starts at midnight am Sunday and ends midnight am Tuesday.
I don't think it's included out the box in dagster - it would be very helpful though
m
As in, a TimeWindowPartitionsDefinition with a weekday
cron_schedule
0 0 * * 1-5
?
b
I'm not sure - are the windows contiguous in that case? I can't tell from the docs
I think this won't cover your use case though - the partitions won't exist for weekends
I think I was wrong earlier about this being the solution to your question though. I'm not quite sure how to set that up with dagster.
m
I just found this from looking into
TimeWindowPartitionsDefinition
, Dagster team's reply was this, so maybe it'll work, I'll have to test it out
Copy code
To have custom time window partitions, you can use the TimeWindowPartitionsDefinition class and set its cron_schedule argument to define how the windows should be evaluated.

For example:

Yearly: TimeWindowPartitionsDefinition(cron_schedule="0 0 1 1 *")
Weekdays: TimeWindowPartitionsDefinition(cron_schedule="0 0 * * 1-5")
Quarterly: TimeWindowPartitionsDefinition(cron_schedule="0 0 1 */3 *")
Hm
cron_schedule="0 0 * * 1-5"
actually runs into an error
dagster._check.CheckError: Failure condition: 0 0 * * 1-5 does not support minute_of_hour/hour_of_day/day_of_week/day_of_month arguments
, but a daily cron expression
cron_schedule="0 7 * * *"
works šŸ˜•
Does anyone from Dagster team have insight on ^this error for trying to create a weekdays
TimeWindowPartitionsDefinition
?
šŸ‘€ 1
s
@Mary Hu IIUC you want to keep a single partition per day for your asset but run a schedule on a different cadence. ā€¢ Monday materializes 3 partitions: Sat, Sun, Mon ā€¢ Tu/Wed/Th/Fr just materialize their own partition Is that right?
m
@sean yes! We want to run on weekdays but materialize for all days, where the Monday run handles the weekend data
s
I think you should be able to do this with a regular schedule-- the schedule eval function can return multiple run requests. This is pseudocode but something like this:
Copy code
@asset(partitions_def=DailyPartitionsDefinition(...))
def foo():
    ...

@schedule(weekday_cron_expr)
def foo_schedule(context):
    if is_monday(context.partition_key):
        return [RunRequest(...) for day in [last_sat, last_sat, today]
    else:
        return RunRequest(...)  # just today
m
oh I see, I didn't know a schedule execution can run multiple partitions, I'll test it out, thanks!
that worked, thanks!
šŸ™Œ 1