Hi I am trying the new DBT API and I have an issue which is dagster #integration-dbt

Hi, I am trying the new DBT API and I have an issu...

Timothee Vandeput

07/10/2023, 7:10 PM

Hi, I am trying the new DBT API and I have an issue which is related to how we structure our dbt project. Our sources have the same name as our models (raw layer). So we have the following dbt unique id : • Source : 'source.our_target.name_of_model' • Model : 'model.our_target.name_of_model' This is not an issue for dbt but when I want to run the model only (subset selection), dagster is not returning the correct fqn for the output name. The issue is with the function that map output names to dbt unique ids. It's only taking the last part of the unique id and thus returning the node info from the source instead of the model. I had no issue with the legacy dbt api. Would it be possible to fix that issue ?

🙏 1

Timothee Vandeput

07/10/2023, 7:51 PM

I fixed the issue by overriding the method node_info_by_output_name of DbtManifest and filtering out sources from the dictionnary :

Copy code

class CustomDbtManifest(DbtManifest):
    @property
    def node_info_by_output_name(self) -> Mapping[str, Mapping[str, Any]]:
        """A mapping of the default output name for a dbt node to the node's dictionary representation in the manifest.
        """
        return {node["unique_id"].split(".")[-1]: node for node in self.node_info_by_dbt_unique_id.values() if node['resource_type'] != 'source'}

Dbt sources shouldn't generate any output anyway so it shoudn't be an issue.

👍🏽 1

rex

07/10/2023, 7:58 PM

We can address this — mind filing an issue?

Timothee Vandeput

07/10/2023, 9:05 PM

Here it is : https://github.com/dagster-io/dagster/issues/15207

rex

07/11/2023, 5:17 PM

Just closing this thread as well, should be resolved with https://github.com/dagster-io/dagster/pull/15226.

👌 1

2 Views

Open in Slack

Previous Next