19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi, is there a way to create partitions (for OPs o...
# ask-communityi
Ismael Rodrigues
05/27/2023, 5:26 AMHi, is there a way to create partitions (for OPs or Assets) considering only business days?
c
claire
05/30/2023, 5:08 PMHi Ismael. I wrote up a short discussion to answer this question: https://github.com/dagster-io/dagster/discussions/14528
🌈 1
i
Ismael Rodrigues
05/30/2023, 7:50 PMgotcha, didn't know I could use DynamicPartitions with OPs
Ismael Rodrigues
05/30/2023, 7:50 PMthanks
Ismael Rodrigues
05/30/2023, 7:58 PMDoes MultiPartitions work with OPs as well?
c
claire
05/30/2023, 9:16 PMYes, you can specify a MultiPartitions def as the partitions definition on an op job at the job level
🌈 1
i
Ismael Rodrigues
05/31/2023, 3:23 PMhttps://github.com/dagster-io/dagster/discussions/14528
I don't know if I got the right effect from this, but when I used It had no difference at all, all the partitions were there from 2023-01-01 to yesterday, even not working days as Saturday and Sunday :/
Ismael Rodrigues
05/31/2023, 3:25 PMmaybe here where you defined DailyPartitionsDefinitions, should be the business_days defined before
(edit) I tested myself, and got an error lmao, so this isn't the case
Ismael Rodrigues
05/31/2023, 5:44 PMWhat I need to do is: have a Monthly schedule that runs a job on the first day of the month, only if it is a working day and isn't a holiday
Example: Runs 01/06/2023, but don't run if it's not working day and then alter the run to tomorrow, until it can run on a working day
c
claire
05/31/2023, 8:17 PMHi Ismael. With the code example, I meant that whatever assets that should have business day partitioning need to use the
DynamicPartitionsDefinition(name="business_days")❤️ 1
claire
05/31/2023, 8:20 PMhave a Monthly schedule that runs a job on the first day of the month, only if it is a working day and isn't a holidayYou could define a custom schedule for this, that evaluates every day. Every day it can check for the first day of the month that is a working day and not a holiday. If that selected day is the currently executing day, it can yield a run request, skipping otherwise.
❤️ 1
i
Ismael Rodrigues
05/31/2023, 10:07 PMThe code you used can be an example of a Custom Schedule? If isn't, is there any place I can see a example of how to build my own?
c
claire
05/31/2023, 10:09 PMAh, my code example builds a schedule from a partitioned job. Here's an example of a custom schedule, where you can run extra logic: https://docs.dagster.io/concepts/partitions-schedules-sensors/schedules#schedules-that-provide-custom-run-config-and-tags
As you can see in the example, you can fetch the scheduled execution time datetime object. You can use this to determine the current month. If you have a function that returns the first valid day in the month, you can check if the current day is the first valid day and yield a run request.
❤️ 1
i
Ismael Rodrigues
05/31/2023, 10:45 PMthanks, I'll take a look
Ismael Rodrigues
06/01/2023, 6:04 AMIf my memory doesn't fail me, I remember at some point having the possibility to create a partition configuration based on a function_fn which returns a list of DateTime or Strings that will be used as the partitions for that job, I searched the entire documentation but today the only class that has this is DynamicPartitions but as the documentation says, will be removed on 2.0.0
I managed to create a function that returns a list of dates containing only the first working day of every month, but I wanted to use it on a Partitioned job, If I can't do this anymore, I'll use DynamicPartition sensor build_add_request option.
c
claire
06/01/2023, 5:33 PMYep, the function that returns a list of partition keys is the older implementation of
DynamicPartitionsDefinition🌈 1
i
Ismael Rodrigues
06/01/2023, 6:31 PMThanks, I'll try do create the dynamic partition using the sensor approach, but if I can't I'll use this one 🙂
Ismael Rodrigues
06/02/2023, 3:24 AMIs there any way I can schedule a dynamic partitioned job?
Ismael Rodrigues
06/02/2023, 3:24 AMI tried build_schedule_from_partitioned_job, but I get an error
c
claire
06/02/2023, 5:33 PMbuild_schedule_from_partitioned_job@schedule🌈 1
i
Ismael Rodrigues
06/06/2023, 6:47 PMbut how the heck will I configure this schedule?
I mean, there's only cron config to use on this custom schedule and how'd they work with Dynamic partition?
I'd need to configure a cron schedule to run every X hours and define if it'll release a run request or not using if date in generate_partitions() then run, else skip?
62 Views