The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hello, is there a bug in Dagit’s partitions for `define_asset_job` on dbt assets? the partition count is not working (showing 0) despite having the partition_defs passed. When I pass empty assets to the job instead of dbt, they show up.

image.png

run_ods has dbt assets and shows up as 0 partitions as seen in screenshots

Looks like it’s caused by the number of assets - this job has 140 and individual assets show the partitions. Using an asset group with 6 assets works fine.

Hi Peter. Does the correct partition count eventually load on the `run_ods` page? I think it displays as 0 until fully loaded, and sometimes this page can be slow when many assets exist in the job.

Hi Claire, it doesn’t but I’ll try leaving it open for a while - going to try using load_assets_from_dbt_manifest as well and see if it solves the problem

^load_assets_from_dbt_manifest doesn’t work.. I might try to use an op-based job to have it render. Thanks!

Do you have a `partitions_def` defined in `define_asset_job`?

Yep, I have these as my define_asset_job. ODS has 140+ models and shows up with 0 partitions, while DWH has 6 models and shows up with the proper partitions
```run_dwh_job = define_asset_job("run_dwh_job", config=dbt_client_config, partitions_def=client_partitions_def,selection=AssetSelection.groups("dwh"))

run_ods_job = define_asset_job("run_ods",config=dbt_client_config, partitions_def=client_partitions_def,selection=AssetSelection.groups("ods"))```

Hm... locally I was able to find that a job with a partitions def but containing only unpartitioned assets displays as having 0 partitions. Is it possible this is the case for you? Can you validate that your `dwh` assets have partitions defs?

Hi <@U028M11QNDD> sorry I didn’t see your reply! Yup dwh (70 models after building out) have partition defs and they show up in the partitions page. ODS have the same configuration but do not show up (coz 140+ models, but dropping it down to 70 makes it show up).

I’m really curious as to how to handle this. 140+ models will defeat the purpose of observability in the partitions page. Is there a way to group these models? i.e still run those 140 models but only observe a few categories? Like if those models are pre grouped and only the grouping categories shows as success/failure in the partitions page

Thank you!!