https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
s

Szymon Piskorz

02/10/2023, 9:48 AM
Hello, I am facing a problem when loading the dbt manifest:
Copy code
UserWarning: Error loading repository location repo.py:dagster._core.errors.DagsterInvalidDefinitionError: Assets can only depend on themselves if they are time-partitioned and each partition depends on earlier partitions
Is there a way to go about this or does it require before DBT compilation action?
r

rex

02/10/2023, 2:43 PM
Do you have a cyclic dependency in your dbt project? cc @sandy since this looks like an interaction with asset partition dependencies
s

Szymon Piskorz

02/15/2023, 9:05 AM
@rex I don’t think so. I even analysed it with networkx and checked if it had cycles and it did not. Although we have about 120MB of manifest file size. I would love to get some guidence when it comes to how to load projects of this size and some troubleshooting guidence. So far we had never succeded with loading the manifest during our PoC but we are interested in using Dagster on larger scale.
@Bartosz Kopytekcc
b

Bartosz Kopytek

02/15/2023, 12:48 PM
Fixed with:
Copy code
dbt_assets = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR, node_info_to_asset_key = lambda node_info: AssetKey(["dbt"] + (node_info["unique_id"].split("."))))
Would love to see docs being more elaborate on this, especially when migrating big DBT projects to Dagster.
11 Views