19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hello, our automaterialize daemon has suddenly sta...
# dagster-plusl
Leo Qin
08/03/2023, 10:48 PMHello, our automaterialize daemon has suddenly started launching repeated runs of the same four assets... it looks it's starting about 4-5 runs at the same time. I do see that the
asset_evaluation_idLeo Qin
08/03/2023, 10:50 PMone of the assets in that set is failing, so basically every two minutes we get a set of 5 failure notifications. The assets in the run are all newly deployed
o
owen
08/03/2023, 11:04 PMHi @Leo Qin -- we're looking into this issue and trying to diagnose the root cause. Looks like you've turned off the daemon for the time being, we'll let you know when the issue is resolved and it's safe to turn back on
🌈 1
l
Leo Qin
08/03/2023, 11:41 PMIf it helps, we figured out the root cause of the run failures in the failing assets - the failing asset depends on another asset that was updated in the same deployment. It's just a code version change, so automaterialize did not kick off that asset. We re-materialized that asset with the new code manually, and the failing assets were able to succeed on a manual run as well.
o
owen
08/03/2023, 11:44 PMHi @Leo Qin -- we were able to track down the root cause of the repeating runs issue. It's actually a pre-existing issue that crops up in the circumstances of:
1. an asset is added with a lazy AMP + no history of materialization
2. its first execution fails
3. it has at least one non-source asset dependency and at least one source asset dependency
owen
08/03/2023, 11:46 PMI just posted a PR for a fix, but it sounds like you should be in the clear for now, as you were able to successfully materialize your assets (which should stop the cycle of materializations)
l
Leo Qin
08/03/2023, 11:48 PMah, thanks for looking into it!
Leo Qin
08/03/2023, 11:48 PMreactivated the automaterialize daemon, will observe for a bit
o
owen
08/03/2023, 11:57 PMgot it -- let me know if you see anything out of the ordinary
18 Views