Hi, I'm trying to turn a normal asset that's been a bottleneck into a partitioned asset to improve performance. The problem is this asset depends on multiple non-partitioned upstream assets. When I try to run the job, it's launching runs for each partition (as expected), but each of these runs seems to be independently generating all of the upstream assets and blowing up resource usage. What I want is each upstream asset to just be created once, then used by all of the partitions. Is there a way to achieve this behavior that I haven't been able to figure out, or should I be using a different pattern here? I've also considered creating a factory function to create individual assets for each partition. There's not too many partitions, so this seems like probably a fine solution, but I wanted to see if I could get the built-in asset partition functionality working as desired first.

Hi Zach, Each run is scoped to at most one partition. We are working to get asset memoization/staleness working with partitions, but for now your best bet is to materialize the non-partitioned assets in one run and then use a separate asset job to launch-partition-scoped runs (they should still be able to load the unpartitioned upstream inputs even if these assets are not included in the run)

Ah ok, that makes sense, thanks!

Thanks @sean! Is there a github issue or discussion we could contribute to?

This discussion does not exactly match the topic, but it’s a reasonable place to post any thoughts on staleness: https://github.com/dagster-io/dagster/discussions/13102 Feel free to be as expansive as you like in your wishlist! Btw I am point-person on data versions and staleness.

I’ve tried to explain our use case and prior expectations about how it would work in this comment: https://github.com/dagster-io/dagster/discussions/13102#discussioncomment-5548988