19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Is there a plan to support FreshnessPolicies for a...
# ask-communityp
Pablo Beltran
04/03/2023, 1:20 AMIs there a plan to support FreshnessPolicies for assets with MultiPartitionsDefinition in the near future?
s
sandy
04/03/2023, 4:09 PMHere's the issue where we're tracking this: https://github.com/dagster-io/dagster/issues/13219
We don't have immediate plans for this, but it's something we'd like to get to soon, i.e. within the next month or two
sandy
04/03/2023, 4:09 PMOut of curiosity, are you using freshness policies for scheduling, alerting, or both?
p
Pablo Beltran
04/03/2023, 4:11 PMI have multi partitioned DBT assets that i would like to schedule using a reconciliation sensor.
Pablo Beltran
04/03/2023, 4:12 PMI am thinking of using multi assets sensor to get the same behavior as having these assets on a 0 second freshness policy
s
sandy
04/03/2023, 4:23 PMdoes that mean that your goal is basically to re-materialize the asset partitions as soon as upstream partitions change? if so, you should be able to accomplish that without FreshnessPolicies, just by using
build_asset_reconciliation_sensorp
Pablo Beltran
04/03/2023, 4:45 PMYeah that's what I was trying to do but i couldn't get it to work so I assumed it was because it needed a freshness policy. I set it up for an asset group that has many deps that were not ready. However one of the assets in the group had all of it's deps ready and it did not kick off a run. Is that because all the deps need to be ready before any runs start?
Pablo Beltran
04/03/2023, 4:45 PM(would making a sensor per asset fix things)
s
sandy
04/03/2023, 4:52 PMThat sounds unexpected that it didn't work - if you're able to provide more details, I could help you figure out why
sandy
04/03/2023, 4:52 PMWe're looking to build a UI that will make it easier to debug these kinds of situations: https://github.com/dagster-io/dagster/issues/13331
p
Pablo Beltran
04/03/2023, 6:20 PMI'll try and get more details for you. And that UI seems very useful!
Pablo Beltran
04/03/2023, 6:46 PMSo here is the asset, as you can see its dep has some partitions that have been materialized successfully and yet it has not kicked off any runs. The asset is part of an asset group called:
'marts__user_metrics' Copy code
USER_METRICS_RECONCILIATION_SENSORS = [build_asset_reconciliation_sensor(
name='user_metrics_reconciliation_sensor',
minimum_interval_seconds=60 * 5,
asset_selection=AssetSelection.groups('marts__user_metrics'),
description='''
This sensor will make it so that and DBT model that is
put under the marts/user_metrics directory will be automatically
run whenever its dependecies are updated.
'''
)]marts__user_metricss
sandy
04/04/2023, 3:47 PMSo here is the asset, as you can see its dep has some partitions that have been materialized successfully and yet it has not kicked off any runs.My suspicion about what's going on is that the asset reconciliation sensor currently doesn't launch runs for partitions that are more than a day old. We made it work like this because many users found it concerning that, when they first turned on the sensor, it would launch an enormous number of runs to fill in old partitions. So you would currently need to run a backfill manually for those partitions. We have a plan to make this customizable in the near future: https://github.com/dagster-io/dagster/discussions/13300.
sandy
04/04/2023, 3:48 PMAlso another weird thing is that when I go to assets that are unrelated to theThis is a silly artifact of how this is all implemented that shouldn't be related to the problem you're facing. We're working on moving asset reconciliation to a daemon instead of a sensor, which should solve this.asset group and look at the sensors (where it says '10 sensors') I see basically all of the reconciliation sensors I have set up for the repo even if they have nothing to do with that asset.marts__user_metrics
10 Views