The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Question about software-defined assets. I understood that it’s possible to create dependencies across assets specified in different code locations. Now my question is, would it be possible to define a dependency on an asset deployed on different dagster instances?

My usecase would be: every team has a dedicated Dagster instance with a group of assets maintained by the team. Team A would like to rematerialize one of their assets whenever an asset from team Z materializes. How could I implement such a solution?

hi <@U02JK6RR5HQ>! this is not directly possible, as the way dagster knows an upstream asset has been updated is by querying the dagster instance (and so if the materialization event is in a different database, it won't have a way of knowing about that).

you could conceivably create a custom sensor that runs in Team A's code location, but queries Team Z's dagster instance, but in general the recommended pattern for multi-team deployments would be to have everyone share a DagsterInstance and just have different code locations

I see. My idea was to have one separate Dagster per team to isolate different teams and provide some access control so that every team can see only the assets/jobs that are owned by them. A single Dagster instance managed by 40/50 teams sounds challenging to navigate.

Is there a way for Dagster to push materialization events to a queue somewhere and then let another dagster process events from this queue? Maybe a combination of hooks and sensors?

<@U01J51Y6B9D> Do you think this might make sense, or maybe there is a better way? :point_up: :thinking_face:
Just trying to understand if my concerns are valid regarding separating different teams in different instances

My personal bias would be to keep everything on a single instance, or at least limit the number of distinct instances (maybe break it up at a coarser grain than per-team). If teams are operating truly independently, then there's no big issue with separating things out, but at the point that you have cross-team dependencies, having a centralized place to represent these dependencies is going to be pretty important

Anecdotally, most Dagster Cloud users tend to keep all of their teams on a single deployment

Thank you for your feedback. Very much appreciated