Nicholas Pezolano
02/22/2023, 3:38 PMScheduleDefinition
that uses should_execute
I created a simple test case below:
from dagster import Definitions, load_assets_from_modules, define_asset_job, AssetSelection, ScheduleDefinition
from . import assets
def test_holiday(context) -> bool:
# returns a boolean (True if the schedule should execute). Defaults to a function that always returns True.
dt = context.scheduled_execution_time # 2023-02-20T16:30:00-05:00
ds = str(dt).split('T')[0]
if ds == '2023-02-20':
print('Should Skip!')
return False
else:
print('Should Execute!')
return True
my_assets = load_assets_from_modules([assets], group_name='my_job_group')
my_job = define_asset_job("my_job", AssetSelection.groups('my_job_group'))
my_schedule = ScheduleDefinition(job=my_job, should_execute=test_holiday, cron_schedule="5 8 * * 1-5", execution_timezone='America/New_York')
defs = Definitions(assets=[*my_assets], schedules=[my_schedule])
and my assets.py file:
from dagster import asset
from dagster import DailyPartitionsDefinition
my_partitions_def = DailyPartitionsDefinition(start_date="2023-01-12",
fmt='%Y-%m-%d',
timezone='America/New_York')
@asset(partitions_def=my_partitions_def)
def test_asset(context):
current_partition_window = context.partition_time_window
print(current_partition_window)
prev_window = my_partitions_def.get_prev_partition_window(current_partition_window.start)
prev_dt = str(prev_window.start).split('T')[0]
print(context.partition_key, prev_dt) #I would expect prev_dt to be 2023-02-17 on 2023-02-21 not 2023-02-20
return prev_dt
1. I'm able to materialize my job for 2023-02-20 even thou should execute returns false for that date. For backfills I would expect 2023-02-20 not to run.
2. The previous partition key for 2023-02-21 shows up as 2023-02-20 should this not be 2023-02-17 in the schedule above.claire
02/24/2023, 10:12 PMshould_execute
returning false simply means that the schedule will skip execution when that is the given day. However, you can still materialize the 2023-02-20 partition (via a run/backfill) as the schedule doesn't gate that behavior.
(2) The asset has a daily partitions definition, which includes all days including weekends. The schedule doesn't gate the existence of the 2023-02-20 partition, so the previous day is 2023-02-20.
If you wanted to make your schedule skip the weekend days, you could define a custom TimeWindowPartitionsDefinition(cron_schedule="...")
object.