Xavier BALESI08/03/2022, 1:58 PM
owen08/03/2022, 7:08 PM
then invoke that generator in the op, but that only handles the input side. The output to be processed by the IOManager has to be emitted all at once (i.e.
def get_generator_function(context): def _fn(): yield 1 return _fn def load_input(self, context): my_fn = self.get_generator_function(context) return my_fn
), so there's no direct support for chunked outputs. However, you could bring the output handling logic inside the body of the asset, and write chunks of your output to either the final location (meaning your IOManager's handle_output would essentially be a no-op), or a temporary file, which the IOManager would then copy over to a final location.
yield Output(output_name="result", value=12345)
Xavier BALESI08/03/2022, 11:24 PM
owen08/04/2022, 4:42 PM