Hi all, thinking about something like the architecture pictured. Basically what I'm thinking is that I want to deploy multiples of dagit that will each handle seperate parts of the data pipeline. so the provider accounts source data from non reproducible data sources, do some deidentifaction and cleaning. next the enrichment accounts act on data from provider accounts and introduce enriched features sets. finally specific projects get access to encriched datasets, possible from multiple sources. in the architecture I would like for the project accounts to be aware when source data has been updated, as well as for the project accounts to get an idea of the exisitng data lineages. as such I want to give project accounts the ability to add user code deployments from the higher level pipelines. does this deployment strategy make sense from dagsters viewpoint (i.e 1-n relationship for usercodedeployments-dagit) and how would I go about allowing the downstream dagit instances know when upstream data changes have been made? Currenlty I understand that all this info would be stored in the sql database and that is tied to dagit so from my current understanding downstream dagits would be in the dark about this. downstream dagit should not have mutate permissions on upstream projects. currently I think this will be handled behind the scenes via IAM so I think that should just work. eg a project user tries to initiate a job in an upstream code deployment they wont have appropriate permissions and it should just fail. sorry bit of a brain dump 😐

does this deployment strategy make sense from dagsters viewpoint

I cant say I’ve ever seen someone do this, but I don’t see a reason why it would not work.

how would I go about allowing the downstream dagit instances know when upstream data changes have been made?

You should be able to use

sensors

to have the downstream listen for upstream changes.

I was thinking more of having the native upstream changed functionality eg in attached picture. I guess all dagit instances could share the same sql server perhaps? not sure what the security implications of that would be