https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alexandre Guitton

04/04/2023, 8:53 AM
Hello 👋, a quick question, I am experimenting with
build_asset_reconciliation_sensor
and
FreshnessPolicy
. I have the following configuration:
Copy code
redshift_sensor = build_asset_reconciliation_sensor(
    asset_selection=AssetSelection.groups(REDSHIFT),
    name="redshift_reconciliation_sensor",
)
and
Copy code
@asset(
    compute_kind="redshift",
    required_resource_keys={"redshift_kpi"},
    freshness_policy=FreshnessPolicy(cron_schedule="45 * * * *", maximum_lag_minutes=60)
)
def data_for_people_kpis(
My current state is as shown in the image above. My question: Why doesn't the sensor trigger materialization for the two stored assets?
v

Vinnie

04/04/2023, 10:03 AM
What’s the freshness policy on your two stale assets? It’s probably something to do with that. Also wanna call attention to this github discussion.
a

Alexandre Guitton

04/04/2023, 10:07 AM
For the two stale assets, there is no freshness policy
v

Vinnie

04/04/2023, 10:11 AM
In this case the behavior I would expect is for the sensor to kick off a run about 60 minutes after the upstream materialization, but I don’t think I have a similar scenario in my infrastructure.
s

sandy

04/04/2023, 3:58 PM
Currently, it's a little weird: because
data_for_people_kpis
has a freshness policy, that affects the materialization schedule of the upstream assets as well. Soon, we're going to make a change that will allow you to set a policy on each asset for how you want it to be automatically materialized: https://github.com/dagster-io/dagster/discussions/13300. This change would allow you to make
hiring_plan_metrics
materialized "eagerly", while
data_for_people_kpis
lazily according to the freshness policy.
a

Alexandre Guitton

04/04/2023, 5:34 PM
Oh nice 😮 it is exactly what I was looking for dagster bot responded by community I have another brain block related to using this feature and it may be related to the definition of the concept of
Late
. Let's assume that my Gsheets are my sources and my Redshifts are my DTMs. The behavior I want is that every day at noon, I import the Gsheets to S3 and then build the Redshift tables. And I have an SLA that says my Redshift table has to be available at 8am or I'm late. In my mind, my definition of
late
for my SLA is not the same
late
as my freshness policy, right? As for the asset,
late
means more like it's time to update the asset if it's stale, right?
s

sandy

04/05/2023, 12:28 AM
The freshness policy is essentially the SLA. We would have named it SLA, but that word has a lot of baggage. Once the changes I linked are merged, freshness policies will only be involved in scheduling if you set your
AutoMaterializationPolicy
to have
for_freshness=True
. If you do that, then Dagster will try to make it so that the assets complete by 8 am. If you want them to start at noon, then you'd probably want to use a
ScheduleDefinition
to trigger them at noon.
Does that make sense?
a

Alexandre Guitton

04/05/2023, 7:04 AM
Um, maybe I missed something, but how is Dagster going to know how long it takes for assets to be finished by 8am? I mean, let's say I have this SLA for a hundred tables that produce my marts, I say for all of those tables that I want them to be finished by 8am and the whole computation for my assets takes about 6 hours, how is Dagster going to know that it needs to trigger the run at 2am, or will it run it at 8am and in that case my completion time will be 2pm?