Hey yall, we are looking to leverage something sim...
# ask-community
z
Hey yall, we are looking to leverage something similar to: https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#monitoring-daily-assets-to-materia[…]ly-asset-using-a-multi_asset_sensor in order to handle hourly -> daily dependencies I'm curious if there are any performance considerations here? This example assumes a single upstream asset and a single downstream asset. In a real world implementation this would probably grow to be NxM (N = upstreams, M = downstreams). Do we actually need to check each downstream asset if they all exist as part of the same job?
s
@claire - thoughts on whether there's a more efficient way to express this or whether the current scheme is likely to have performance issues? I think
all_partitions_materialized
might do some caching in the background for when two assets share an upstream asset?
c
Hmm... so we actually don't do the caching for
all_partitions_materialized
at the moment. But I think we should, so we won't need to make this query again for calls with the same asset key/partitions. As far as querying for the latest materialization per asset per partition, we do have indexes set up to make these queries fairly performant.
z
Cool - we will plan to utilize this sensor then. Will let ya'll know if we run into any issues
👍 1