https://dagster.io/ logo
#ask-community
Title
# ask-community
t

Tom Reilly

07/15/2022, 3:39 PM
Going off of this, if dagster does not support jobs spanning assets from different repos, are asset sensors currently the best approach to get around this? I'm a little confused what the point of defining SourceAssets are if the dagit lineage is unable to materialize them together and jobs cannot be defined using them. I currently have a repo that has
Copy code
asset1 (source asset from another repo) --> asset2 --> asset3
and was hoping to do something like
Copy code
my_job = define_asset_job(name="my", selection="asset1*")
but that does not appear to work
y

yuhan

07/15/2022, 9:35 PM
cc @sandy
s

sandy

07/15/2022, 9:42 PM
Yes - asset sensors are currently the best approach. We're interested in adding the capability for jobs to span multiple repos in the future. Out of curiosity, what's the reason your assets live in different repos? Happy to find some time to talk through what you're building if it would be helpful
t

Tom Reilly

07/18/2022, 2:33 PM
@sandy We have a split environment between dbt and a legacy in house data pipeline tool. These two envs have different dependencies. We have some pipelines that span the two environments and previously orchestrated via a jenkins job with multiple stages. We are looking to orchestrate in dagster so seemed like best option was to have these two environments setup as 2 repos in the same workspace. However, since jobs cannot span repos it seems like we cannot use
define_asset_job
. I think sensors will work fine for us and if/when cross repo jobs become a feature we can drop the use of sensors
s

sandy

07/18/2022, 3:31 PM
Thank Tom - very helpful!
3 Views