https://dagster.io/ logo
Title
z

Zach Schira

03/30/2023, 4:28 PM
Hi, I'm trying to turn a normal asset that's been a bottleneck into a partitioned asset to improve performance. The problem is this asset depends on multiple non-partitioned upstream assets. When I try to run the job, it's launching runs for each partition (as expected), but each of these runs seems to be independently generating all of the upstream assets and blowing up resource usage. What I want is each upstream asset to just be created once, then used by all of the partitions. Is there a way to achieve this behavior that I haven't been able to figure out, or should I be using a different pattern here? I've also considered creating a factory function to create individual assets for each partition. There's not too many partitions, so this seems like probably a fine solution, but I wanted to see if I could get the built-in asset partition functionality working as desired first.
:dagster-bot-resolve: 1
s

sean

03/30/2023, 4:32 PM
Hi Zach, Each run is scoped to at most one partition. We are working to get asset memoization/staleness working with partitions, but for now your best bet is to materialize the non-partitioned assets in one run and then use a separate asset job to launch-partition-scoped runs (they should still be able to load the unpartitioned upstream inputs even if these assets are not included in the run)
z

Zach Schira

03/30/2023, 4:35 PM
Ah ok, that makes sense, thanks!
b

Bennett Norman

03/30/2023, 6:51 PM
Thanks @sean! Is there a github issue or discussion we could contribute to?
s

sean

03/30/2023, 7:13 PM
This discussion does not exactly match the topic, but it’s a reasonable place to post any thoughts on staleness: https://github.com/dagster-io/dagster/discussions/13102 Feel free to be as expansive as you like in your wishlist! Btw I am point-person on data versions and staleness.
:dagster: 1
z

Zane Selvans

04/07/2023, 12:33 AM
I’ve tried to explain our use case and prior expectations about how it would work in this comment: https://github.com/dagster-io/dagster/discussions/13102#discussioncomment-5548988