Hello! I'm trying to use IOManager to save files into folders. Is there anyway I could send some dynamic filename into IOManager? They are only defined at runtime (for example, the filename should be the result of some calculation), so I can't define it in the pipeline resources.

Hi @Laura Moraes - the only information that's shared by

handle_output

and

load_input

is the information inside the

OutputContext

. So you have a couple options: • Run the same calculation inside both

handle_output

and

load_input

to determine the file path. • Store the calculated file path in some location that both

handle_output

and

load_input

know about. For that, you could do something like wrap the

PickledObjectS3IOManager

to store the file path.

I'm adding @João Luiz Carabetta to this chat

I have some custom IO managers I wrote that do similar to what @sandy described - will post to github when I get a chance. The part I wanted to be able to do but haven't yet is to timestamp the filename but because the config is recalculated on each run that doesn't work if a pipeline is partially rerun.

Hi @sandy, could we talk on Thursday at 9am, pacific time? We were reading the Output documentation, and would like to understand if we could attach these data to the metadata (https://docs.dagster.io/_apidocs/solids#dagster.Output) to pass this information to the IO Manager.

9 am on thursday works. If you DM me your email, I can send you a calendar invite with a zoom link

I was exploring how to get Output metadata in an IO Manager this morning, that would explain why I couldn’t find it. It would be great to be able to access to output metadata, as the information that you may want to encode in a path may not be known before execution.