Any advice on using assets when processing thousands of (mostly)-independent work items through the pipeline? Currently we have ops with DynamicOuts to do a big fan-out across the work items. But internally we do some state management to figure out when we need to re-run different steps for each item, and that sounds like assets. Different item IDs sounds like partitions, but we would only know the partitions after querying the db to list out all the items.

You could look into using DynamicPartitions for the ids! https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions#dynamically-partitioned-assets

Nice, thanks! Hah looking for the right place in my notes to squirrel this advice away, I see that I asked roughly the same question before and you also answered it, but it seems like there are new cooler tools for it now.

yeah dynamic partitions are relatively new but at first glance it seems like a good fit for this!