https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Mark Fickett

05/17/2023, 5:18 PM
Any advice on using assets when processing thousands of (mostly)-independent work items through the pipeline? Currently we have ops with DynamicOuts to do a big fan-out across the work items. But internally we do some state management to figure out when we need to re-run different steps for each item, and that sounds like assets. Different item IDs sounds like partitions, but we would only know the partitions after querying the db to list out all the items.
j

jamie

05/17/2023, 6:10 PM
m

Mark Fickett

05/17/2023, 6:15 PM
Nice, thanks! Hah looking for the right place in my notes to squirrel this advice away, I see that I asked roughly the same question before and you also answered it, but it seems like there are new cooler tools for it now.
j

jamie

05/17/2023, 6:15 PM
yeah dynamic partitions are relatively new but at first glance it seems like a good fit for this!
👍🏻 1
4 Views