https://dagster.io/ logo
#ask-community
Title
# ask-community
i

Ismael Rodrigues

05/27/2023, 5:26 AM
Hi, is there a way to create partitions (for OPs or Assets) considering only business days?
c

claire

05/30/2023, 5:08 PM
Hi Ismael. I wrote up a short discussion to answer this question: https://github.com/dagster-io/dagster/discussions/14528
🌈 1
i

Ismael Rodrigues

05/30/2023, 7:50 PM
gotcha, didn't know I could use DynamicPartitions with OPs
thanks
Does MultiPartitions work with OPs as well?
c

claire

05/30/2023, 9:16 PM
Yes, you can specify a MultiPartitions def as the partitions definition on an op job at the job level
🌈 1
i

Ismael Rodrigues

05/31/2023, 3:23 PM
https://github.com/dagster-io/dagster/discussions/14528 I don't know if I got the right effect from this, but when I used It had no difference at all, all the partitions were there from 2023-01-01 to yesterday, even not working days as Saturday and Sunday :/
maybe here where you defined DailyPartitionsDefinitions, should be the business_days defined before (edit) I tested myself, and got an error lmao, so this isn't the case
What I need to do is: have a Monthly schedule that runs a job on the first day of the month, only if it is a working day and isn't a holiday Example: Runs 01/06/2023, but don't run if it's not working day and then alter the run to tomorrow, until it can run on a working day
c

claire

05/31/2023, 8:17 PM
Hi Ismael. With the code example, I meant that whatever assets that should have business day partitioning need to use the
DynamicPartitionsDefinition(name="business_days")
partitions def. The schedule is just used to create new dynamic partitions on business days.
❤️ 1
have a Monthly schedule that runs a job on the first day of the month, only if it is a working day and isn't a holiday
You could define a custom schedule for this, that evaluates every day. Every day it can check for the first day of the month that is a working day and not a holiday. If that selected day is the currently executing day, it can yield a run request, skipping otherwise.
❤️ 1
i

Ismael Rodrigues

05/31/2023, 10:07 PM
The code you used can be an example of a Custom Schedule? If isn't, is there any place I can see a example of how to build my own?
c

claire

05/31/2023, 10:09 PM
Ah, my code example builds a schedule from a partitioned job. Here's an example of a custom schedule, where you can run extra logic: https://docs.dagster.io/concepts/partitions-schedules-sensors/schedules#schedules-that-provide-custom-run-config-and-tags As you can see in the example, you can fetch the scheduled execution time datetime object. You can use this to determine the current month. If you have a function that returns the first valid day in the month, you can check if the current day is the first valid day and yield a run request.
❤️ 1
i

Ismael Rodrigues

05/31/2023, 10:45 PM
thanks, I'll take a look
If my memory doesn't fail me, I remember at some point having the possibility to create a partition configuration based on a function_fn which returns a list of DateTime or Strings that will be used as the partitions for that job, I searched the entire documentation but today the only class that has this is DynamicPartitions but as the documentation says, will be removed on 2.0.0 I managed to create a function that returns a list of dates containing only the first working day of every month, but I wanted to use it on a Partitioned job, If I can't do this anymore, I'll use DynamicPartition sensor build_add_request option.
c

claire

06/01/2023, 5:33 PM
Yep, the function that returns a list of partition keys is the older implementation of
DynamicPartitionsDefinition
. I think that implementation currently works with op jobs, but not with asset jobs. We do plan on removing it in 2.0.0, but 2.0.0 won't happen for the foreseeable future
🌈 1
i

Ismael Rodrigues

06/01/2023, 6:31 PM
Thanks, I'll try do create the dynamic partition using the sensor approach, but if I can't I'll use this one 🙂
Is there any way I can schedule a dynamic partitioned job?
I tried build_schedule_from_partitioned_job, but I get an error
c

claire

06/02/2023, 5:33 PM
build_schedule_from_partitioned_job
only works for time-partitioned jobs. You would have to define a custom
@schedule
to schedule a dynamically partitioned job
🌈 1
i

Ismael Rodrigues

06/06/2023, 6:47 PM
but how the heck will I configure this schedule? I mean, there's only cron config to use on this custom schedule and how'd they work with Dynamic partition? I'd need to configure a cron schedule to run every X hours and define if it'll release a run request or not using if date in generate_partitions() then run, else skip?
27 Views