How can I alert on asset freshness and send a slack message if an asset is NOT fresh?

hi @geoHeil! There's actually an out of the box sensor for this behavior being added to

dagster-slack

in today's release. Essentially:

make_slack_on_freshness_policy_status_change_sensor(
    asset_selection=..., # assets to alert on
    channel="#foo",
    slack_token=os.getenv("SLACK_API_TOKEN"),
)

What is the difference between https://docs.dagster.io/master/concepts/partitions-schedules-sensors/asset-sensors#freshness-policy-sensors- `

freshness_policy_sensor

and

make_slack_on_freshness_policy_status_changed_sensor

is the Slack one only a special sub-kind of the first one?

@geoHeil That sensor is a

freshness_policy_sensor

, but saves you from having to write the slack-specific code yourself (with some extra options thrown in on top). I accidentally typo'd the name (

changed

should have been

change

), edited above. It's in the

dagster-slack

library version

0.17.6

, and you can find the API docs here: https://docs.dagster.io/_apidocs/libraries/dagster-slack#dagster_slack.make_slack_on_freshness_policy_status_change_sensor

so does this mean that for now if the upstream asset is created based on an event-based sensor that it will not work?

@geoHeil yep that can work! the freshness calculation is independent of how any upstream assets are materialized. It sounds like

FreshnessPolicy(maximum_lag_minutes=???, cron_schedule="0 9 * * *")

would basically work for you. this says "by 9AM, there should be a materialization of this asset which incorporates all upstream data from at least

???

minutes ago". the main friction with the current implementation (vs. the ideal version-based implementation) is selecting a good value for

???

. Right now, this will depend on when you expect the file to come in. If you set maximum_lag_minutes to be too low (let's say

60

minutes), then if the file comes in before that time (let's say 7AM) and everything runs as expected, then the current logic will interpret your downstream asset as "has all of the upstream data up until

7AM

". When 9AM rolls around, the freshness policy will demand that the asset has all the data from up until 60 minutes before 9AM (so

8AM

), meaning it will fail its freshness check, even though everything went as expected.

sounds great.

that's an interesting idea -- I think the tools are there to build that sort of sensor out of the default sensor machinery (there are some internal methods to calculate how late any given asset materialization was with respect to its freshness policy, so you could query for the last N materialization and get their lateness values and come up with some threshold based off of that), but yeah it'd be pretty challenging to do that on your own at the moment

indeed - I think so