Will an Asset Reconciliation Sensor trigger a materialization of downstream assets every time any of its upstream assets is materialized?

import time

@asset
def upstream_short():
   time.sleep(1)
   return 1

@asset
def upstream_long():
   time.sleep(600)
   return 2

@asset
def downstream(upstream_short, upstream_long):
   return 3

reconciliation_snesor = build_asset_reconciliation_sensor(
    asset_selection=AssetSelection.keys("downstream"),
)

I would expect that

reconciliation_sensor

will trigger a run of

downstream

if

upstream_short

is materialized, or if

upstream_long

is materialized. What if

upstream_short

is materialized, while

upstream_long

is running?

Hi Stephen, currently the asset reconciliation sensor doesn't have any functionality to wait depending on whether upstream assets are currently executing. So if upstream long is running and upstream short is materialized, the downstream asset will rematerialize. Then, once upstream long is materialized, the downstream asset will rematerialize again.

ok, thanks. What if I set a longer sensor lag time, like allowing 60 minutes between evaluations? In that case, it would detect both changes, and submit only one run, right?

Yes that’s right, it’ll work as long as the evaluation interval is longer than the greater runtime of the two assets

Hi @claire Do you know if there will be any work planned to add intelligence to the reconciliation sensor to "plan" around the case that Stephen mentioned above? I'm thinking about a scenario where a large backfill is taking place for an upstream of both

upstream_short

and

upstream_long

above called

upstream_main

. In this case

downstream

would materialize twice for each partition

Hi Zachary. We have discussed this and we likely will add a functionality in the future that waits for in progress runs. The main reason why it doesn't exist right now is that it's a less reliable way of waiting for upstream work for partitioned assets. Partitioned executions are often split across many runs that aren't in progress at the same time. There's also the potential concern that it may cause a deadlock where a job may be running constantly. Once we handle these cases, we'll provide support for waiting for in progress runs.

I'm thinking about a scenario where a large backfill is taking place for an upstream of both

upstream_short

and

upstream_long

above called

upstream_main

. In this case

downstream

would materialize twice for each partition

In this case, the existing asset reconciliation sensor should already work fine After

upstream_main

completes,

upstream_long

will be considered stale. The reconciliation sensor won't materialize any assets until all their parents are fresh. I.e. it won't run

downstream

until

upstream_short

and

upstream_long

have both incorporated the change to

upstream_main

.