https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
s

Sandeep Mullangi

02/09/2024, 7:08 AM
Hi Team, I am working on POC on dagster and I am creating downstream table from 1 partitioned table and 1 non-partitioned table. I want my downstream table to trigger if all the upstream partitioned tables have updated data (in this case 1 table). I came up with automaterialization policy. I think non-partitioned table is causing problem here and not allowing downstream to be triggered.
Copy code
auto_materialize_policy = AutoMaterializePolicy.eager(
                max_materializations_per_minute=100

            ).with_rules(
                AutoMaterializeRule.skip_on_not_all_parents_updated(require_update_for_all_upstream_partitions=True)
            ).without_rules(
                AutoMaterializeRule.skip_on_parent_missing(),
                AutoMaterializeRule.skip_on_parent_outdated()
            )
I have freshness policy as backup so that my downstream triggers atleast after time specified and it is also not working. i have following freshness policy
Copy code
freshness_policy = FreshnessPolicy(maximum_lag_minutes=1)
My downstream model says
overdue
I think precedence is given to materialization policy (1 of the skip condition is met) and then freshness is not triggered because of skip condition is met. In this case then what is the use of freshness policy if it is following materialization strictly?
i saw this and looks like my issue is related to this https://github.com/dagster-io/dagster/issues/14422