Should this asset refresh ASAP? I set a `Freshness...
# ask-community
c
Should this asset refresh ASAP? I set a
FreshnessPolicy
that it should materialize by midnight daily, and an
AutoMaterialize.eager
policy. I'm not seeing any runs initiate
here's the fp definition
Copy code
FreshnessPolicy(
    maximum_lag_minutes=0,
    cron_schedule="0 0 * * *",
    cron_schedule_timezone=LOCAL_TIMEZONE.name,
)
c
Hi Charlie. Are the upstream assets up to date? The upstream assets must be up to date in order for the downstream asset to be materialized via an auto-materialize policy
c
Ah so it’s a root asset. Will automaterialize + freshnesspolicy not trigger a run like that?
I set up a schedule instead, but was trying out the two to see if you could do that declaratively
c
Hm... if it's a root asset, i'm surprised that the asset didn't materialize immediately? cc @owen / @johann
o
hm it definitely should materialize immediately -- has this asset been materialized at all before, or would this be the first materialization? and is it partitioned?
c
It would be the first materialization, and is partitioned
The partitions are a custom fiscal year one I created subclassing TimeWindow
o
ah I see -- creating custom PartitionsDefinitions is not supported at the moment (high level: a lot of the logic that works with partitions happens in places where it would be difficult / non-performant to execute arbitrary user code, so these subclasses need to survive a serde boundary). it's possible/likely that this is the source of the issue (very likely the auto materialize code is not understanding your partitions definition) would it be possible to rework your custom subclass into a straight up TimeWindowPartitionsDefinition by setting some combination of
cron_schedule
,
fmt
, and
start_offset
? happy to talk through what that might look like if you have your custom subclass code handy
also, in theory, you shouldn't need a freshness policy here at all -- for time-partitioned assets at the root of the graph, the daemon will just materialize the partitions as soon as they come into existence. the freshness policy in this case would just give you an indicator of if it's taking longer than expected to do so
c
@owen yeah, it’s effectively a wrapper around a
TimeWindowPartitionsDefinition
, so it won't work as expected even if the inherits and runs the
__init__
from that? Here's the code for it: https://github.com/TEAMSchools/teamster/blob/28d1ca49992de6d0ac2438bbcf058acebacb80f7/src/teamster/core/utils/classes.py#L36-L55
my use case is that there's a report we pull that's filtered by fiscal year but I want to make sure we pull a fresh version of it daily with the most up-to-date data, hence the
FreshnessPolicy
o
ah ok -- that subclass implementation should work fine, as you're right it's just setting properties on the class. in this case, I think I'd stick with a schedule implementation for now. regardless of other behaviors, the auto materialize policies will only materialize each partition a single time for root assets (as it assumes that once a partition is filled, it has all the data it needs, and won't revisit it)