Agon Shabi
06/22/2021, 4:52 PMOutputManager
for some "unconnected" outputs - that is, outputs generated at the end of my pipelines that should go to external systems, and definitely won't be used as inputs to downstream solids.
I'd like to avoid writing an IOManager
whose load_input
implementation just raises a NotImplementedError
, this docstring suggests this used to be a thing (?) :
https://github.com/dagster-io/dagster/blob/8b8c658674b63ff88c0ffa641bdedcded3772d0[…]b/python_modules/dagster/dagster/core/storage/output_manager.py
What's the best way to proceed? "result-pusher" solids which delegate to "result-pusher" resources?jordan
06/22/2021, 5:59 PMOutputManagers
were migrated to IOManagers
in the 0.10.0 release:
https://github.com/dagster-io/dagster/commit/3397d9c1eda139c81501d1157a3f72f9e9a334dc
If you want Dagster’s execution to automatically invoke handle_output
, then you’ll need to indeed subclass an IOManager
and not implement load_input
. If I understand your use case correctly, I think that’s the best way to proceed.
But your other suggestion would work too - you don’t strictly need to lean on Dagster automatically invoking handle_output
and you could instead write your own resource that your solids call directly.Agon Shabi
06/23/2021, 10:24 AM