luk
03/08/2023, 1:12 PMVinnie
03/08/2023, 1:18 PMfs_io_manager
emits the absolute path as metadata (see screenshot example)Vinnie
03/08/2023, 1:19 PMluk
03/08/2023, 1:32 PMdef get_metadata(
self,
context: OutputContext, # pylint: disable=unused-argument
obj: Any, # pylint: disable=unused-argument
) -> Dict[str, MetadataValue]:
"""Child classes should override this method to add custom metadata to the outputs."""
return {}
Daniel Gafni
03/08/2023, 2:48 PMluk
03/08/2023, 2:55 PMdef dump_to_path(self, context: OutputContext, obj, path: UPath):
# Note: this is a workaround to have the path available when overriding the get_metadata method
self.path = path
...
since dump_to_path is called first in handle_output I have it available as soon as get_metadata is called. this way i do not need to override too much.Daniel Gafni
03/08/2023, 3:52 PM