https://dagster.io/ logo
#dagster-plus
Title
# dagster-plus
l

Leo Qin

08/03/2023, 10:48 PM
Hello, our automaterialize daemon has suddenly started launching repeated runs of the same four assets... it looks it's starting about 4-5 runs at the same time. I do see that the
asset_evaluation_id
tag is incrementing. Can someone help us understand what is going on? Seemed to start after out latest deployment
one of the assets in that set is failing, so basically every two minutes we get a set of 5 failure notifications. The assets in the run are all newly deployed
o

owen

08/03/2023, 11:04 PM
Hi @Leo Qin -- we're looking into this issue and trying to diagnose the root cause. Looks like you've turned off the daemon for the time being, we'll let you know when the issue is resolved and it's safe to turn back on
🌈 1
l

Leo Qin

08/03/2023, 11:41 PM
If it helps, we figured out the root cause of the run failures in the failing assets - the failing asset depends on another asset that was updated in the same deployment. It's just a code version change, so automaterialize did not kick off that asset. We re-materialized that asset with the new code manually, and the failing assets were able to succeed on a manual run as well.
o

owen

08/03/2023, 11:44 PM
Hi @Leo Qin -- we were able to track down the root cause of the repeating runs issue. It's actually a pre-existing issue that crops up in the circumstances of: 1. an asset is added with a lazy AMP + no history of materialization 2. its first execution fails 3. it has at least one non-source asset dependency and at least one source asset dependency
I just posted a PR for a fix, but it sounds like you should be in the clear for now, as you were able to successfully materialize your assets (which should stop the cycle of materializations)
l

Leo Qin

08/03/2023, 11:48 PM
ah, thanks for looking into it!
reactivated the automaterialize daemon, will observe for a bit
o

owen

08/03/2023, 11:57 PM
got it -- let me know if you see anything out of the ordinary
12 Views