The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Is it in no way possible to create a job that consists of assets in multiple repositories?

I have created a repository approximately per data source (having about 4 sources with a bunch of assets in each), however that means I have a different repo for assets that takes something else as input. Now i want to trigger the whole flow from ingress to egress. Am i missing something or is the splitting of multiple repositories without reason? Should we just have one

I know we should switch to declarations instead of repositories. it's on our roadmap

we don’t have a native way to create one job that consists of asset in multiple repos or code locations, but you can achieve via asset sensor which can operate cross repos: <https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#defining-an-asset-sensor>

the main reason why it’s not possible today is that all steps in a job will be executed in one same environment, whereas different repos (or code locations) can have different python execution environment. so we can’t have a single job for assets across repos.

the asset sensor workaround is essentially splitting the execution into two jobs that are in two repos