When running a two dagster repositories using gRPC servers, dagit properly shows the assets for both and their connections (one repo calls out SourceAssets with matching keys from the other). But when I try to materialize the assets that are downstream of the other repo, they fail because the upstream IOManager in the other repo is not being used downstream. I would have expected the downstream asset to fail because the IOManager upstream was not declared in the downstream repo but the failure I get is that the pickled file can't be found that the default fs_io_manager uses (instead of the SQL IOManager from the upstream repo being called out as not existing). How do you have upstream IOManagers in other repos get used in downstream assets in their own repo process? I am reading up on this more and went down this path based on this comment in the SourceAsset documentation section: 'You can also re-use assets across repositories by including them as source assets'. But it seems to me that in order for this to work the SourceAsset has to have an IOManager configured on it to properly pull the upstream data even if the upstream asset has an IOManager on it also. I am guessing that there is no synchronization mechanism to pull the configuration from an upstream IOManager down to the downstream IOManager in another repo without manually configuring them. Is this true? And is so is there a recommended way to do this?

Hi Chris. Resources exist within the scope of a single repository, so there currently isn't a way to synchronize the configuration of IO managers on a cross-repository basis. The easiest way of handling this case is likely to define an IO manager that can read from the source asset within the downstream repository, and point to this IO manager via the

io_manager_key

or

io_manager_def

params of

SourceAsset

.

@claire Thanks for the quick response!

Hey @claire, do you think it would be possible to make the IOManagers from other repos available to the current repo? It’s seems a big PITA to duplicate IOManagers code across different repositories. Of course you can make a common package or git submodule, but it’s not fun too. I’m currently trying to connect two repositories and noticed this. The same applies to partition definitions I think.

Hi Daniel--sorry about the late response, I was out of office. I think there's not a great way to do this at the moment, I think the easiest way might be to use the common package as you mentioned, importing the IO manager from one repository to the other

Yep I went with this approach