Rob Sicurelli
05/16/2023, 2:34 PMVitaly Markov
05/16/2023, 4:42 PMC
running every 15 minutes.
2. Attach schedule function checking materialization status of asset B
using Dagster instance.
3. If asset B
is "stale" (was materialized longer than XXX minutes ago), yield SkipReason
from schedule. Otherwise, yield RunRequest
.
An example of how to fetch last mat. event: https://github.com/dagster-io/dagster/discussions/14311AssetSensor
will not work, since it uses "cursor" internally and can only "consume" materialization event once.Rob Sicurelli
05/16/2023, 5:28 PMsandy
05/17/2023, 5:18 AMAutoMaterializePolicy(on_missing=True, on_new_parent_data=True, for_freshness=False, time_window_partition_scope_minutes=datetime.timedelta.resolution.total_seconds() / 60)
. A might still execute though? I'm not 100% sure.
• You could add an observable source asset upstream of C and have a schedule observe it every 15 minutes and give C an eager auto-materialize policy.Rob Sicurelli
05/17/2023, 6:08 PMsandy
05/17/2023, 7:13 PMRob Sicurelli
05/17/2023, 9:26 PMdagster-dbt
? I only see this in dagster_dbit/asset_utils.py
def default_auto_materialize_policy_fn(
node_info: Mapping[str, Any]
) -> Optional[AutoMaterializePolicy]:
auto_materialize_policy = node_info["config"].get("dagster_auto_materialize_policy", {})
if auto_materialize_policy.get("type") == "eager":
return AutoMaterializePolicy.eager()
elif auto_materialize_policy.get("type") == "lazy":
return AutoMaterializePolicy.lazy()
return None
I think if the mapping from DBT accepted those custom arguments, we could just create a custom AutoMaterializePolicy
in that method on line 91. happy to submit a PR to add this. chatting with my team it's behavior that we would use broadly, since we don't want expensive assets to get triggered incidentally. ideally they own their own materialization logicsandy
05/18/2023, 8:51 PMload_assets_from_dbt_manifest
accepts a node_info_to_auto_materialize_policy_fn
that allows you to basically provide a custom implementation of that function
we also welcome a PR, but that might move a little slower as we iron out the exact names we want to use, etc.Rob Sicurelli
05/18/2023, 10:52 PM