Hello folks! I am having some trouble setting up ...
# ask-community
j
Hello folks! I am having some trouble setting up the
AutoMaterializePolicy
for my use case. I have the following case:
Copy code
A --> B --> C
A, B and C have
AutoMaterializePolicy.eager()
A and B are partitioned C is not partitioned C depends on B, using
deps=[B]
. When I trigger A, B is automatically materialized, as expected. However, C is never materialized... I just want C to be triggered every time B triggers, regardless of which partition was triggered. I know I can create a sensor, but I'd need to create lots of sensors because we have multiple places with the same setup Any workarounds/suggestions?
🤖 1
j
Also interested in this use case
j
Sorry for upping this... But I'd really like to know if there is any workarounds or a proper way of doing this. This is blocking our upgrades to dagster 1.4
o
Hi @João Meyer -- in the Auto-materialize History tab (on the asset details page for asset C), are you able to see the past evaluations for C? My assumption is that C is being skipped because some partitions of A are newer than the corresponding partitions of B (and so the daemon is waiting for that to be corrected before proceeding with materializing C). This behavior isn't always desirable, and we're working on allowing users to turn off this check (and get the true "whenever parent is updated, update" behavior).
j
Thanks for your answer @owen I tested it again by manually materalizing A and waiting for B to auto materialize I kept checking C's auto materialize history tab and I saw the following: 1. No conditions met 2. Skipped a. Materialization conditions met i. Materialization is missing ii. Upstream data has changed since latest materialization (link points to asset B) iii. Required to meet this asset's freshness policy iv. Required to meet a downstream freshness policy b. Skip conditions met i. Waiting on upstream data (link points B) 3. No conditions met No runs were triggered between steps 1, 2 and 3 for asset C. After everything, C does not materialize and keeps showing the "upstream data" icon above the asset
Maybe we are missing a check for the case where the upstream asset that changed is the same one we are waiting on? Or we are missing rechecks after asset B finishes execution? Which would not meet the skip condition
I am trying to find in the code where this logic is applied. But I a not sure where it is. Any tips?
o
I can see how the way that's currently displayed is confusing, but in this case the "waiting on upstream data (link points to B)" means that the daemon believes that asset B is out of date with respect to its parent (asset A, in this case).