@claire, fantastic! I think it would fix my problem entirely. 🔥
I am working on declarative job configs and alternative scheduling implementation. It goes quite well so far.
Instead of having thousands of multi-asset sensors (one per job), I create one super-sensor for many jobs at once.
This sensor pre-loads last materialization for all assets and counters for all asset partitions.
After that sensor iterates over assets and checks the status of upstream dependencies. For partitioned assets, it checks every non-materialized downstream partition and compares it with status of corresponding upstream partitions.
If all upstream partitions are materialized in all parents, but downstream partition is not yet materialized -> yield RunRequest.
The main goal is to scale well and manage dependencies for very large number of assets and jobs. Also, I add some custom scheduling logic and conditions which go beyond current AutoMaterializePolicy / FreshnessPolicy.