19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
How can I retrieve the metadata which is set in on...
# ask-communityg
geoHeil
05/18/2022, 3:35 PMHow can I retrieve the metadata which is set in one op/asset like:
context.add_output_metadata({'key': 'value'})context.get_logged_metadata_entries()s
sandy
05/18/2022, 6:07 PMThere's currently not a straightforward way to do this, though it wouldn't necessarily be difficult to add one. @Alex Kerney raised a comprehensive proposal that included this, which I unfortunately haven't had the chance to give the attention it deserved: https://github.com/dagster-io/dagster/discussions/6913.
Out of curiosity, what are you interested in using this for?
a
Alex Kerney
05/18/2022, 6:17 PMI also haven’t gotten nearly as much time to think further on this, but I would be really interested in hearing other’s thoughts too.
g
geoHeil
05/18/2022, 8:03 PMthis is related to https://dagster.slack.com/archives/C01U954MEER/p1652600973766459 basically I want to be able to feed the dynamic partition (as a tag or additional metadata) somehow to the IO manager as well as to a downstream cleaning step
geoHeil
05/18/2022, 8:29 PMThanks for sharing this ticket! @Alex Kerney I agree with all your points - and feel the sam pain.
geoHeil
05/19/2022, 4:21 PMI am experimenting with the approach outlined by @Alex Kerney but stumble upon multiple questions/issues: https://github.com/dagster-io/dagster/discussions/6913#discussioncomment-2784327 and https://github.com/dagster-io/dagster/discussions/6913#discussioncomment-2784399 and https://github.com/dagster-io/dagster/discussions/6913#discussioncomment-2784887. so far, I have managed to output dynamic partitions suggested by the approach of Alex. But I am still trying to figure out how to be able to read them in (for a downstream task).
When assuming two assets:
- ingestion
- cleaning
ingestion could set a tag/metadata to the materialization
cleaning could by default handle the latest tag
But then 2 questions remain for me:
1) assuming both assets are in the same job/dag/graph how can the 2nd one be triggered in case a fresh dynamic partition is created?
2) expecting failures to happen (i.e. requirements of backfilss) how could someone straight from dagit UI overwrite the notion of latest to trigger any other run (i.e. assuming a backfill of the first ingestion job which requires also re-running of the 2nd cleaning one for each of the outputted dynamic partitions)
6 Views