Also, as a feature suggestion, it would be nice if you could specify an output io manager at a graph level. Right now I have a graph that builds a dataframe. I want that dataframe saved to a database, and I wrote a custom io manager to do that. But I have no way to attach it to the output of the graph. Instead, I have to attach it to the output of the terminal op of the graph, which feels inelegant. That op is an extremely generic piece and I don't think it should care about the final destination of its product.
06/17/2022, 11:32 PM
Hi Dmitry. In dagster, graphs are wrappers that construct the dependencies between different ops. Because graphs can be reused in different jobs, we don't allow binding graphs to specific resources (e.g. IO managers).
I think for your use case where the terminal op is reused across multiple jobs, it might help to have a per-output IO manager. In different jobs where the op is reused, you can provide a different IO manager for the same resource key.
06/17/2022, 11:37 PM
using a single key name that can be specialised for different jobs is a really good suggestion. I'll do that.