Lorenzo
07/25/2022, 1:15 PMload_assets_from_dbt_project
.
Is it possible to distinguish each dbt model based on the tag that is associated to? For example: i tag only one model with the tag "finance" and in dagster I want to run only the specific model who has the "finance" tag, which could be only one, or even more than one.sean
07/25/2022, 4:16 PMowen
07/25/2022, 4:23 PMload_assets_from_dbt_project(..., select='tag:finance')
. The select argument here takes in a dbt-syntax select statement, and will limit the models you load to only ones matching that statement. Another option, if you don't have overlapping tags (i.e. each model will only ever have at most a single tag) would be to map the tag of the model to dagster's "group" concept. You can do this by setting the node_info_to_group_fn
(https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.load_assets_from_dbt_project) argument to something like lambda info: info["tags"][0]
(didn't test that, but it'll be that basic shape). This way, each dbt asset loaded will be mapped to a group based on its first tag, and you can select from those groups using AssetSelection.groups("finance")
Lorenzo
07/26/2022, 9:20 AMJames Hale
10/08/2022, 3:42 PMnode_info_to_asset_key
parameter of load_assets_from_dbt_project()
.Lorenzo
01/13/2023, 10:00 AMnode_info_to_asset_key
but I can't understand how to use it 🥲. My idea was to import all the assets at once (with one statement, like this: dbt_assets = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR)
, but still have the control over one single dbt model at a time, so that I could restart the dag "from point of failure" without re-materializing also the upstream models that succeded.
How should I set up the "node_info_to_asset_key" to do so? dbt_assets = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR, node_info_to_asset_key= [ ???? ])
Just to give some context: at the moment I successfully reached my goal by just importing one model at a time, but the project is getting exponentially big and models are now hundreds 👀
# Load the assets from your dbt project automatically using models names
dbt_assets_1 = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR, select="first_dbt")
dbt_assets_2 = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR, select="second_dbt")
dbt_assets_3 = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR, select="third_dbt")
dbt_assets_4 = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR, select="fourth_dbt")
dbt_assets_5 = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR, select="fifth_dbt")
dbt_assets_6 = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR, select="sixth_dbt")
dbt_assets_7 = load_assets_from_dbt_project(project_dir=DBT_PROJECT_DIR, select="seventh_dbt")