https://dagster.io/ logo
#ask-community
Title
# ask-community
n

Nicholas Pezolano

02/22/2023, 3:38 PM
I noticed a few unexpected behaviors with a
ScheduleDefinition
that uses
should_execute
I created a simple test case below:
Copy code
from dagster import Definitions, load_assets_from_modules, define_asset_job, AssetSelection, ScheduleDefinition

from . import assets

def test_holiday(context) -> bool:
    # returns a boolean (True if the schedule should execute). Defaults to a function that always returns True.
    dt = context.scheduled_execution_time # 2023-02-20T16:30:00-05:00
    ds = str(dt).split('T')[0]
    if ds == '2023-02-20':
        print('Should Skip!')
        return False
    else:
        print('Should Execute!')
        return True

my_assets = load_assets_from_modules([assets], group_name='my_job_group')
my_job = define_asset_job("my_job", AssetSelection.groups('my_job_group'))

my_schedule = ScheduleDefinition(job=my_job, should_execute=test_holiday, cron_schedule="5 8 * * 1-5", execution_timezone='America/New_York')

defs = Definitions(assets=[*my_assets], schedules=[my_schedule])
and my assets.py file:
Copy code
from dagster import asset

from dagster import DailyPartitionsDefinition

my_partitions_def = DailyPartitionsDefinition(start_date="2023-01-12",  
                                            fmt='%Y-%m-%d',
                                            timezone='America/New_York')

@asset(partitions_def=my_partitions_def)
def test_asset(context):
    current_partition_window = context.partition_time_window
    print(current_partition_window)
    prev_window = my_partitions_def.get_prev_partition_window(current_partition_window.start)
    prev_dt = str(prev_window.start).split('T')[0]
    print(context.partition_key, prev_dt) #I would expect prev_dt to be 2023-02-17 on 2023-02-21 not 2023-02-20
    return prev_dt
1. I'm able to materialize my job for 2023-02-20 even thou should execute returns false for that date. For backfills I would expect 2023-02-20 not to run. 2. The previous partition key for 2023-02-21 shows up as 2023-02-20 should this not be 2023-02-17 in the schedule above.
c

claire

02/24/2023, 10:12 PM
Hi Nicholas. The schedule executing is independent of the asset materializing. (1) for 2023-02-20,
should_execute
returning false simply means that the schedule will skip execution when that is the given day. However, you can still materialize the 2023-02-20 partition (via a run/backfill) as the schedule doesn't gate that behavior. (2) The asset has a daily partitions definition, which includes all days including weekends. The schedule doesn't gate the existence of the 2023-02-20 partition, so the previous day is 2023-02-20. If you wanted to make your schedule skip the weekend days, you could define a custom
TimeWindowPartitionsDefinition(cron_schedule="...")
object.
2 Views