When materialising multiple partitions in a single...
# ask-community
b
When materialising multiple partitions in a single run, is there a way to provide output metadata on a per-partition basis? (Possibly related: the GUI reports 'When graphing values by partition, the highest data point for each materialized event label is displayed.' - I'm not entirely clear on what that means.)
j
hey @Brendan Jackson i don’t think this is possible since only one output is returned for the execution, even though multiple partitions are being filled. I’m not entirely sure what that text in the UI means either, i’ll bring it up with the team
👍 1
b
hey Brendan - I’m the author of the text above 🙂 The note means that if you materialize the same partition several times and we have more than one MaterializationEvent providing a metadata value
abc
for partition A, the graphed value shown for that partition is
MAX(abc)
. Will see if we can make that more clear!
b
Hi Ben - thanks! That makes sense for things like run times. But if the metadata is something like row count of an asset, that's much less helpful!
So I'm clear, does this mean any metadata on a partition is always going to be the max of all the metadata ever generated for that partition? (For
abc
, in your example)
Even for things like run times, without any expiry this seems unhelpful. Is there a reason not to just choose the latest value for that partition?