Aki Iwa
04/18/2023, 6:24 AMmodels:
modelA: # project name
modelA:
.....
modelB:
+schema: B
.....
Asset is automatically loaded in Data Lineage as follows.
• data asset
model/modelA/model_a_table
model/modelB/model_b_table
I am having trouble linking the modelB side and the materialization of the Data Lineage, although I am doing dbt runs daily and materializing at the same time as follows.
dbt_output = context.resources.dbt.run()
for materialization in generate_materializations(dbt_output):
yield materialization
This is because the “node” in the dbt_output output results in model.modelA.model_a_table, model.project.model_b_table
and therefore will be materialized as model.modelA.model_b_table.
Is there a good way to tie the materialization of the results from the daily dbt run to the data lineage?Aki Iwa
04/18/2023, 6:25 AMowen
04/18/2023, 4:25 PMload_assets_from_dbt_(project/manifest)
and a custom dbt run op. In most cases, this is not necessary, as you can directly create a job to materialize the assets you've loaded with load_assets_from_dbt...
. There are some examples of that here: https://docs.dagster.io/integrations/dbt/reference#scheduling-dbt-jobs, but the basic idea is that you can create a job to materialize some or all of your defined assets using define_asset_job("my_job", selection=...)
.
This means you can remove the op/job where you're calling
dbt_output = context.resources.dbt.run()
for materialization in generate_materializations(dbt_output):
yield materialization
and instead rely on the built-in implementation provided by the asset integration