https://dagster.io/ logo
#ask-ai
Title
# ask-ai
s

Sterling Paramore

03/11/2024, 9:49 PM
Can I build a job that contains assets from multiple code locations?
s

Scout

03/11/2024, 9:49 PM
Based on the documentation, a job is typically associated with a single code location, which is a collection of Dagster definitions loadable and accessible by Dagster's tools. Each code location comprises a reference to a Python module that has an instance of
Definitions
in a top-level variable and a Python environment that can successfully load that module. Definitions within a code location have a common namespace and must have unique names. However, the documentation does not explicitly state whether a job can contain assets from multiple code locations. It does mention that code locations are loaded in a different process and communicate with Dagster system processes over an RPC mechanism, which suggests that there might be some architectural considerations when attempting to build a job that spans multiple code locations. For more complex asset definitions and dependencies across code locations, the documentation suggests using
SourceAsset
to represent external assets that are generated elsewhere, allowing other assets to depend on them. This could potentially be used to reference assets across different code locations, but the documentation does not provide a clear example of a job that directly contains assets from multiple code locations. Given this information, if you are looking to build a job that targets assets in different code locations, you may need to consider the architecture of your Dagster deployment and how code locations are managed. It might be necessary to use
SourceAsset
to reference assets from other code locations or to structure your code in a way that all required assets are within the same code location for the job. If you need further clarification or guidance on this topic, I would recommend checking the latest Dagster documentation or reaching out to the Dagster community for more specific use cases and examples.
3 Views