Up until this point my team has used a single code location for our Dagster (cloud) deployment. We're starting to feel some growing pains around managing dependencies in one big pyproject.toml file / environment, so I'm starting to investigate splitting up our deployment into multiple code locations. I know there used to be some limitations around references across code deployments but am unsure how things have changed (if at all) with Definitions and whatnot. Could someone highlight some of the limitations of having multiple code locations? I'm particularly interested in things like the following:
• can you refer to assets in a different code location when building asset jobs or declaring asset dependencies?
• can sensors in one code location reference a job in a different location?
• does querying the event logs from the DagsterInstance retrieve event logs across all code locations in a deployment?
• any other cross-code location limitations that I should be aware of?
• separate code locations can define different run launchers, correct?
dagster bot responded by community 1
03/21/2023, 9:58 PM
I don't really see how definitions would change this, since AssetKeys are just an event log query. So, to answer your 3rd question, I believe that as long as they're pointing at the same dagster database, then queries should be ok. Which means that 1 & 2 work as well.
(disclaimer: we're still using repositories to run multiple code locations from the same deployment, but definitely have asset dependencies and sensors working between code locations)
03/21/2023, 10:02 PM
okay cool that's great, thanks for the info!
03/21/2023, 10:52 PM
Querying From a dagit ui that has a workspace with one repository will still show runs from other repositories