https://dagster.io/ logo
#ask-community
Title
# ask-community
r

R Lucas

03/08/2023, 7:44 PM
Hello, I'm currently running the latest version (1.1.21) and facing issue with backfill of time-partitionned assets: the very first partition is always viewed as "missing" despite being properly materialized in a given previous run. All of my time-partitioned seem to have this issue with their first partition.
s

sandy

03/08/2023, 11:32 PM
hey Lucas - would you be able to share code that reproduces this issue?
r

R Lucas

03/09/2023, 7:58 PM
Hello Sandy, I just replicated the issue with the following code
Copy code
from dagster import (
    HourlyPartitionsDefinition,
    asset,
    Definitions
)

hourly_partition = HourlyPartitionsDefinition(start_date="2023-03-09-15:00")

@asset(partitions_def=hourly_partition)
def hourly_asset():
    pass

defs = Definitions(assets=[hourly_asset])
See result below
Hello, Any idea on this issue ?
s

sandy

03/10/2023, 5:50 PM
sorry for the delay on this. I just gave this a try, but alas, was not able to reproduce. here's what I did: • ran a backfill over all the partitions (26) in your example - observed that they all ended up green • tried adjusting the start date so that there would be just 3 partitions, to match your screenshot - observed that they all ended up green is it possible that one of the runs failed in your case?
r

R Lucas

03/10/2023, 7:16 PM
Hello Sandy, For your information, I was able to get a proper result downgrading to version 1.1.19 but the issue happens from version 1.1.20.
Running my time-partition assets on version 1.1.20 and newer produces the issue: first partition is identified as missing despite being produced. I am able to see materialized asset for the "missing" partition so I know it was not a failed run.
s

sandy

03/11/2023, 5:39 PM
interesting - I was running on master
I am able to see materialized asset for the "missing" partition so I know it was not a failed run.
where exactly are you seeing that?
r

R Lucas

03/11/2023, 5:43 PM
Hello Sandy, I tested using IO manager writing assets on my local FS and can see output. As shown in the image the run exists and is completed successfully. The only issue is this partition greyed both on partition bar and when looking at asset list page. This first partition is also still shown as missing on backfill menu.
Hello @sandy, I'm stiil facing this issue running Dagster on my laptop but I just found that defining time partitions with my timezone (CET) rather than leaving default timezone (UTC) solves the issue : all partitions are properly detected. Is this expected behavior ?
s

sandy

03/14/2023, 12:37 AM
Hmm - that is very a useful clue. @claire - would you mind looking into this tomorrow?
c

claire

03/14/2023, 4:26 PM
I will take a look at this today
Fix: https://github.com/dagster-io/dagster/pull/12944 The cause is that the partitions def start time is timezone-aware, but a validation check to see if a partition key occurred before the start time created datetime objects that were not timezone aware (defaults to timezone of running OS). So this error occurs when the user's OS is ahead of the partitions def timezone (by default UTC).
r

R Lucas

03/14/2023, 8:57 PM
Thank you for the update
Hello claire, Is the fix applied on the latest release or will it be on the next ones ?
c

claire

03/17/2023, 4:51 PM
It will be applied in next week's release
2 Views