https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Daniel Gafni

08/07/2023, 7:38 PM
AutoMaterializePolicy
creates a Run for 5 assets from which 2 are currently being materialized - is this normal? It shouldn't be, right?
o

owen

08/07/2023, 10:26 PM
hi @Daniel Gafni! can you share a bit more about the shape of the graph you're working with / which assets were getting updated and which were current materializing?
e

Erek Kiiker

08/08/2023, 12:10 PM
Hi, We have experienced the same behavior with our setup. Asset is being materialized and while it still running a new run is created.
d

Daniel Gafni

08/08/2023, 1:17 PM
A -> B B -> (C, D) (B, C) -> E A gets materialized by a sensor (C, D) is a
@graph_multi_asset
o

owen

08/08/2023, 6:34 PM
What's the timeline here? Is it something like: 1. A is materialized 2. run is kicked off for B, C, D, E 3. a duplicate run is kicked off for B, C, D, E? Or are there other events happening in the middle here? One thing to note is that there is (currently) no special handling for cases where an asset is in progress, all else being equal. I.e. if you have a simple graph
A -> B
, and A takes 1 second to complete, while B takes 30 minutes, and B has an eager policy, then materializing A the first time will of course kick off a run of B, but each materialization of A before that first run of B completes will also kick off a run of B. It sounds like that might not be the semantics you're expecting here, is that fair?
n

Nikolaj Galak

08/08/2023, 9:25 PM
@owen, I observe following behavior: 1. Asset graph is sequential: A -> B -> C -> D -> E 2. E has
Freshness policy
defined and Lazy
AutoMaterializePolicy
, rest of assets don't have policies 3. Assets A, C, D, E are fast, B is long running 4. Run is triggered for assets A, B, C, D, E based on Freshness policy 5. A is materialized 6. While B is being materialized, another run is triggered for C and D only, no extra runs for E It looks like
AutoMaterializePolicy
is not considering planned materializations
o

owen

08/09/2023, 7:01 PM
@Nikolaj Galak I'm guessing in this case, the other assets also have AutoMaterializePolicies, they just don't have freshness policies (otherwise the daemon would never kick off a run for A B C and D), is that accurate? Also, are there any source assets in your setup? The policy is definitely intended to take planned materializations into account, but a bugfix is going out in this week's release that patches up a case where that wasn't happening properly (involving assets that had both source asset parents and non source asset parents)
n

Nikolaj Galak

08/09/2023, 7:16 PM
@owen, yes, you are right, lazy
AutoMaterializePolicy
is set for all the assets, not just E.
A
is source asset and
B, C, D, E
are dbt assets in my example. Great to hear that bugfix is on its way, appreciated!
o

owen

08/09/2023, 8:01 PM
yep -- hard to tell if this is exactly the same case, but definitely let me know if things seem resolved after the latest release, and if not we can dig in more!
e

Erek Kiiker

09/01/2023, 6:00 AM
Hi @owen, we are still observing the behavior where another run of asset with AutomaterializePolicy on. First run finished about 5 seconds after second run was started. We are running version 1.4.10
4 Views