https://dagster.io/ logo
#ask-community
Title
# ask-community
g

geoHeil

05/18/2022, 3:35 PM
How can I retrieve the metadata which is set in one op/asset like:
context.add_output_metadata({'key': 'value'})
in an IO manager?
context.get_logged_metadata_entries()
returns an empty list for me
s

sandy

05/18/2022, 6:07 PM
There's currently not a straightforward way to do this, though it wouldn't necessarily be difficult to add one. @Alex Kerney raised a comprehensive proposal that included this, which I unfortunately haven't had the chance to give the attention it deserved: https://github.com/dagster-io/dagster/discussions/6913. Out of curiosity, what are you interested in using this for?
a

Alex Kerney

05/18/2022, 6:17 PM
I also haven’t gotten nearly as much time to think further on this, but I would be really interested in hearing other’s thoughts too.
g

geoHeil

05/18/2022, 8:03 PM
this is related to https://dagster.slack.com/archives/C01U954MEER/p1652600973766459 basically I want to be able to feed the dynamic partition (as a tag or additional metadata) somehow to the IO manager as well as to a downstream cleaning step
Thanks for sharing this ticket! @Alex Kerney I agree with all your points - and feel the sam pain.
I am experimenting with the approach outlined by @Alex Kerney but stumble upon multiple questions/issues: https://github.com/dagster-io/dagster/discussions/6913#discussioncomment-2784327 and https://github.com/dagster-io/dagster/discussions/6913#discussioncomment-2784399 and https://github.com/dagster-io/dagster/discussions/6913#discussioncomment-2784887. so far, I have managed to output dynamic partitions suggested by the approach of Alex. But I am still trying to figure out how to be able to read them in (for a downstream task). When assuming two assets: - ingestion - cleaning ingestion could set a tag/metadata to the materialization cleaning could by default handle the latest tag But then 2 questions remain for me: 1) assuming both assets are in the same job/dag/graph how can the 2nd one be triggered in case a fresh dynamic partition is created? 2) expecting failures to happen (i.e. requirements of backfilss) how could someone straight from dagit UI overwrite the notion of latest to trigger any other run (i.e. assuming a backfill of the first ingestion job which requires also re-running of the 2nd cleaning one for each of the outputted dynamic partitions)
6 Views