Edson Henrique
04/07/2023, 9:22 PMQwame
04/08/2023, 3:54 AMnode_info_to_group_function
Edson Henrique
04/12/2023, 4:56 PM{{
config(
materialized="incremental",
schema="assets",
tags=["asset", "places", "freshness_sensor", "staging"],
unique_key = 'place_id',
incremental_strategy = 'merge',
node_info_to_group_fn = {
"group_name": "staging"
},
dagster_freshness_policy={
"maximum_lag_minutes": (60 * 7),
"cron_schedule": "*/40 6-23 * * *",
"cron_schedule_timezone": "America/Sao_Paulo"
})
}}
Qwame
04/12/2023, 5:50 PMload_assets_from_dbt_project
, you use the node_info_to_group_fn
to pull that tag as a group name{{
config(
materialized="incremental",
schema="assets",
tags=["asset", "places", "freshness_sensor", "staging"],
unique_key = 'place_id',
incremental_strategy = 'merge',
dagster_freshness_policy={
"maximum_lag_minutes": (60 * 7),
"cron_schedule": "*/40 6-23 * * *",
"cron_schedule_timezone": "America/Sao_Paulo"
})
}}
Then in dagster, (you can also use load_assets_from_dbt_manifest
)
def _node_info_to_group(node_info):
return node_info["tags"][3]
load_assets_from_dbt_project(
project_dir='..',
profiles_dir='..',
select="tag:staging",
node_info_to_group_fn=_node_info_to_group
)
This assumes that in your tags config list, the staging
tag is always the 4th item in the listdbt
will accept additional configs like you have in the dagster_freshness_policy
, I know you can put anything in under meta
. But if dbt accepts dagster_freshness_policy
and group_name
like you have above, then you can have the change the node_info function:
You can inspect the manifest.json
to see how the additional parameters you put under config
to see how it shows in the manifest and then pull it like you will access a python dictionary object. Hope this makes senseEdson Henrique
04/13/2023, 3:59 PM