I m using the experimental API for yielding metadata from an

Question

I m using the experimental API for yielding metadata from an IO Manager I have written two classes decorator pattern MetaDataDecorator and DataframeToCsvDecorator which yield meta data and then delegate the call to an underlying IOManager When using a decorated IOManager return DataframeToCsvDecorator base dir MetaDataDecorator PickledObjectFilesystemIOManager base dir the call chain works when not yielding any meta data but it only results in call to the outer decorator DataframeToCsvDecorator when meta data is yielded Is this an intended behavior can anyone suggest a workaround

owen · Answer

hi < Thorsten Schäfer> do you mind sharing a code snippet here You can remove any business logic just trying to get a better sense of the shape of what you re doing

Thorsten Schäfer · Answer

Sure `class MetaDataDecorator IOManager ` `def init self base io manger IOManager ` `self base io manager = base io manger` `def handle output self context OutputContext obj Any gt None ` `self base io manager handle output context obj ` `if isinstance obj DataFrame ` `asset name = context asset key path 1 ` `context log debug f Storing meta information for data frame output of asset asset name ` ` TODO yielding currently not working` ` yield MetadataEntry value= len obj label= Number of rows ` ` yield MetadataEntry value= len obj columns label= Number of columns ` ` yield MetadataEntry value=MetadataValue text string max rows=10 label= Sample ` `def load input self context InputContext gt Any ` `return self base io manager load input context `

Thorsten Schäfer · Answer

`class DataframeToCsvDecorator IOManager ` `def init self base dir base io manger IOManager ` `self base dir = base dir` `self base io manager = base io manger` `def get path self output context OutputContext gt str ` `return os path join self base dir f output context asset key path 1 csv ` `def handle output self context OutputContext obj Any gt None ` `self base io manager handle output context obj ` `if isinstance obj DataFrame ` `asset name = context asset key path 1 ` `path = self get path context ` `context log debug f Storing CSV for data frame output of asset asset name to path path ` `<http obj to|obj to> csv path index=False ` ` TODO yielding currently not working` ` yield MetadataEntry value=MetadataValue path os path abspath path label= CSV path ` `def load input self context InputContext gt Any ` `return self base io manager load input context `

Thorsten Schäfer · Answer

` io manager ` `config schema= base dir Field StringSource is required=False ` `description= Built in filesystem IO manager that stores and retrieves values using pickling ` ` ` `def csv io manager init context ` `base dir = init context resource config get ` ` base dir init context instance storage directory ` ` ` `return DataframeToCsvDecorator base dir MetaDataDecorator PickledObjectFilesystemIOManager base dir `

Thorsten Schäfer · Answer

When I use the IOManager csv io manager with the decorators using the above code it works As soon as I uncomment the `yield MetaDataEntry` lines in the code only the outer decorator `DataframeToCsvConverter` gets called but the call does not go to the `MetaDataDecorator` and the `PickledObjectFilesystemIOManager`

owen · Answer

got it I see what s happening here when you have a `yield` statement inside of a function it becomes a generator This generator does not fully execute when invoked i e ``` gt gt gt def f yield 1 1 0 yield 2 gt gt gt f lt generator object f at 0x1071736d0 gt gt gt gt ```

owen · Answer

to fix this you can change the first line of handle output to ``` def handle output self context OutputContext obj Any gt None yield from self base io manager handle output context obj ```

owen · Answer

`yield from` will exhaust the generator from the inner IOManager s handle output function before moving on

Thorsten Schäfer · Answer

Thanks < owen> that s working nicely +1