Vitaly Markov05/22/2023, 7:23 AM
, which returns latest mat. event for one partition. And it is affected by backfills, so it is not always the last partition. I see
which is available in asset record. It holds information about partition statuses, but no timestamps. I see
, which is very close, but it returns only
, but not
. In theory, I can use sensor cursor, call
directly and react on materialization events of specific partitions. But I might have "one-to-many" dependencies, when one asset waits for changes in multiple upstream partitions. Just one isolated mat. event is not enough, I am looking for a full picture. I've noticed multi-asset-sensor and clever logic around consumed / unconsumed events there. It might be acceptable in theory, but I am going to have thousands of jobs monitored like this, and dependency graph might change at any time. I am worried about performance and fragility of dependency checks on asset graph changes between sensor evaluations. Ideally, it would be nice to do all checks in one super-sensor or custom daemon using 1-2-3 SQL selects per tick for all assets in repo. Is there anything else which I might have missed? Thank you!
claire05/22/2023, 5:01 PM
per-asset partition. You can try to use the multi asset sensor since it does have built-in methods to fetch the latest unconsumed materialization per partition. The cursor stores partitioning information, so it is possible if your partitions change frequently that you may need to do cursor resets when this occurs. Otherwise, I'd recommend filing an issue for your use case, since I do agree that it would be nice to support an instance method that fetches latest materialization per partition.
Vitaly Markov05/22/2023, 5:56 PM
table with latest mat events per partition (similar to existing
) is almost unavoidable. It is crucial to fully support all edge case scenarios for Auto-Materializing assets. For example: https://github.com/dagster-io/dagster/issues/14385 I'll do some more tests and create an issue in a few days. Thank you!
claire05/22/2023, 6:25 PM
Vitaly Markov05/23/2023, 11:02 AM