19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hello,
I'm currently running the latest version (1...
# ask-communityr
R Lucas
03/08/2023, 7:44 PMHello,
I'm currently running the latest version (1.1.21) and facing issue with backfill of time-partitionned assets: the very first partition is always viewed as "missing" despite being properly materialized in a given previous run. All of my time-partitioned seem to have this issue with their first partition.
s
sandy
03/08/2023, 11:32 PMhey Lucas - would you be able to share code that reproduces this issue?
r
R Lucas
03/09/2023, 7:58 PMHello Sandy,
I just replicated the issue with the following code
Copy code
from dagster import (
HourlyPartitionsDefinition,
asset,
Definitions
)
hourly_partition = HourlyPartitionsDefinition(start_date="2023-03-09-15:00")
@asset(partitions_def=hourly_partition)
def hourly_asset():
pass
defs = Definitions(assets=[hourly_asset])R Lucas
03/10/2023, 5:06 PMHello,
Any idea on this issue ?
s
sandy
03/10/2023, 5:50 PMsorry for the delay on this. I just gave this a try, but alas, was not able to reproduce. here's what I did:
• ran a backfill over all the partitions (26) in your example - observed that they all ended up green
• tried adjusting the start date so that there would be just 3 partitions, to match your screenshot - observed that they all ended up green
is it possible that one of the runs failed in your case?
r
R Lucas
03/10/2023, 7:16 PMHello Sandy,
For your information, I was able to get a proper result downgrading to version 1.1.19 but the issue happens from version 1.1.20.
R Lucas
03/10/2023, 7:18 PMRunning my time-partition assets on version 1.1.20 and newer produces the issue: first partition is identified as missing despite being produced. I am able to see materialized asset for the "missing" partition so I know it was not a failed run.
s
sandy
03/11/2023, 5:39 PMinteresting - I was running on master
I am able to see materialized asset for the "missing" partition so I know it was not a failed run.where exactly are you seeing that?
r
R Lucas
03/11/2023, 5:43 PMHello Sandy,
I tested using IO manager writing assets on my local FS and can see output. As shown in the image the run exists and is completed successfully. The only issue is this partition greyed both on partition bar and when looking at asset list page.
This first partition is also still shown as missing on backfill menu.
R Lucas
03/13/2023, 8:47 PMHello @sandy,
I'm stiil facing this issue running Dagster on my laptop but I just found that defining time partitions with my timezone (CET) rather than leaving default timezone (UTC) solves the issue : all partitions are properly detected.
Is this expected behavior ?
s
sandy
03/14/2023, 12:37 AMHmm - that is very a useful clue. @claire - would you mind looking into this tomorrow?
c
claire
03/14/2023, 4:26 PMI will take a look at this today
claire
03/14/2023, 7:44 PMFix: https://github.com/dagster-io/dagster/pull/12944
The cause is that the partitions def start time is timezone-aware, but a validation check to see if a partition key occurred before the start time created datetime objects that were not timezone aware (defaults to timezone of running OS). So this error occurs when the user's OS is ahead of the partitions def timezone (by default UTC).
r
R Lucas
03/14/2023, 8:57 PMThank you for the update
R Lucas
03/17/2023, 4:50 PMHello claire,
Is the fix applied on the latest release or will it be on the next ones ?
c
claire
03/17/2023, 4:51 PMIt will be applied in next week's release
2 Views