Hi 👋 I am trying to load dbt assets from multiple projects. By reading https://dagster.slack.com/archives/C01U954MEER/p1669307372666059 I could define multiple assets using

dbt_resource_key

argument with

load_assets_from_dbt_projects

. But I am having problem include both dbt asset groups in the repository. I can include either one of them in the return statement of

repository

. But when I include both of them, I get below error.

dagster._core.errors.DagsterInvalidDefinitionError: Invalid dependencies: op "run_dbt_b7d19" does not have input "source_moo_moocards_moocards_moo_user". Available inputs: ['source_moo_recurly_billing_info', 'source_moo_recurly_invoices', 'source_moo_recurly_transactions']

The dbt assets are defined as below.

def get_dbt_group_name(node_info: Mapping[str, Any]):
    name = _get_node_group_name(node_info)
    return "dbt_" + name if name else None


# setup an `asset` with all dbt models
all_models = with_resources(
    # when loading assets from dbt, we invoke the `node_info_to_group_fn` argument
    # to label them all under a unified `group_name`
    load_assets_from_dbt_project(
        project_dir=DBT_PROJECT_DIR, profiles_dir=DBT_PROFILES_DIR, node_info_to_group_fn=get_dbt_group_name
    ),
    {
        "dbt": dbt_cli_resource.configured(
            {"project_dir": DBT_PROJECT_DIR, "profiles_dir": DBT_PROFILES_DIR, "target": DBT_TARGET}
        )
    },
)

# setup an `asset` with dbt prep models
prep_models = with_resources(
    load_assets_from_dbt_project(
        project_dir=DBT_PREP_DIR,
        profiles_dir=DBT_PROFILES_DIR,
        dbt_resource_key="dbt_prep",
        node_info_to_group_fn=get_dbt_group_name,
    ),
    {
        "dbt_prep": dbt_cli_resource.configured(
            {"project_dir": DBT_PREP_DIR, "profiles_dir": DBT_PROFILES_DIR, "target": DBT_TARGET}
        )
    },
)

Could I get some insight on how to load both dbt assets into the repository? Thank you!

hi @Ricky Kim! This is a pretty surprising error -- I played around with this for a bit trying to replicate it but wasn't able to do so. Just to be clear, does this error show up as soon as you try to load the repository? And is there anything within your repository other than those two sets of assets? And are

['source_moo_recurly_billing_info', 'source_moo_recurly_invoices', 'source_moo_recurly_transactions']

the only sources in one of your projects? Or is that an incomplete list? A final question, do you happen to have a target_path set in your dbt_project.yml?

Thank you for your reply. To answer your questions, "does this error show up as soon as you try to load the repository?" When I branch deploy, it will fail to load. And I click into "view the error", I see the error message "And is there anything within your repository other than those two sets of assets?" From the error message

dagster._core.errors.DagsterInvalidDefinitionError: Invalid dependencies: op "run_dbt_b7d19" does not have input "source_moo_moocards_moocards_moo_user". Available inputs: ['source_moo_recurly_billing_info', 'source_moo_recurly_invoices', 'source_moo_recurly_transactions']

['source_moo_recurly_billing_info', 'source_moo_recurly_invoices', 'source_moo_recurly_transactions'] is the complete set from a project stored in DBT_PREP_DIR (in my code snippet). "source_moo_moocards_moocards_moo_user" is just one model from the total 23 models from a project stored in DBT_PROJECT_DIR. My guess is that it's trying to load a model from DBT_PROJECT_DIR, when it should load models from DBT_PREP_DIR. _"And are

['source_moo_recurly_billing_info', 'source_moo_recurly_invoices', 'source_moo_recurly_transactions']

the only sources in one of your projects?"_ Yes this is a complete set of models from DBT_PREP_DIR. "do you happen to have a target_path set in your dbt_project.yml?" Yes I have same target_path set in both projects' dbt_project.yml. as

target-path: "target"

Regarding your suggestion of using

load_assets_from_dbt_manifest

, I will try that and see how it goes. But if you can spot anything from my answer could you kindly let me know? Thank you.

Hi @owen I’ve done a little more digging to try and get to the bottom of things on this one. I did a

pprint(vars(dbtproj_asset[0]))

for both dbt projects of ours in conjunction with the error message we are seeing:

Error loading repository location moo:dagster._core.errors.DagsterInvalidDefinitionError: 
Invalid dependencies: op "run_dbt_b7d19" does not have input "source_moo_moocards_moocards_order_item". 
Available inputs: ['source_moo_recurly_billing_info', 'source_moo_recurly_invoices', 'source_moo_recurly_transactions']

The mentioned

source_moo_moocards_moocards_order_item

will be different each time trying to start dagit (picks up the 1st in whatever ordered list generated on each run of dbt project 1). dbt project 2 only has the 3 sources mentioned in Available inputs. There seems to be something strange going on where it can’t distinguish the difference between the 2 projects - like it’s looking at assets of project 2 with the lens of project 1 The one important thing to note is that the 3 sources named in dbt project 2 also exist in dbt project 1. I.e.

['source_moo_recurly_billing_info', 'source_moo_recurly_invoices', 'source_moo_recurly_transactions']

is common in both projects and appears in the

vars(…)

output under the

_keys_by_input_name

key. The mystery part is that by providing the `include`/`exclude` kwarg for

load_assets_from_dbt_project

, some voodoo happens and there are no complaints and the project loads fine. Does the addition of those flags trigger some sort of different code flow? The

vars(..)

output of the assets with and without the exclude/include shows the identical object, but perhaps internally there is something different?

hey @Manish Khatri! I really appreciate the in-depth digging here -- the fact that the exclude argument is fixing the issue leads me to suspect that this is somehow a name collision issue. Basically, when there are multiple dbt projects being loaded, we need to come up with a unique name for the op that backs the assets for each of those projects. By default, we do this by reading in the project_id from the manifest.json file. Then, we name the op dbt_run_{project_id}. However, as you can see in the line below that, if "select" or "exclude" is specified, then we add in a hash of the select/exclude statement to the op name. So my best guess here is that somehow the project id for each of these projects is identical (or at least the first 5 characters are identical) -- it might be worth checking that on your end, just to know, and so we're creating two ops with duplicate names, which is confusing the system. When you add in an empty exclude statement, it changes the generated name, which solves the collision.

Good shout, I presume

project_id

is the equivalent of the

name:

key in the

dbt_project.yml

? If so, then yes these are identical and we should change this (!)

ah yeah it might be a hash of the name field or something like that!

Ok great, will fix that and see if that resolves this issue! 🤞