https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Daniel Gafni

04/18/2023, 8:04 AM
The asset reconciliation sensor has materialized a downstream asset twice. Why can this happen?
@owen I believe you are working on the sensor?
o

owen

04/18/2023, 9:52 PM
ah yep -- I'm guessing that screenshot is showing all runs for the instance (i.e. there's no other materialization that might have happened in between those two runs)? definitely seems like a bug. do you mind checking how long the earlier run took to execute? also things like when it got dequeued etc. theoretically, the sensor should take in-progress runs into account, but it's possible that something is going wrong there (as a first guess)
also, what assets are upstream of
ar-resp/predictions
? are they partitioned as well? also daily?
d

Daniel Gafni

04/18/2023, 9:54 PM
Daily partitioned source assets -> inference_data (daily partitioned) Non-partitioned source asset + inference_data -> predictions (daily partitioned)
The first run was executed in around 45 minutes. 10 minutes for inference_data and 35 for predictions. It took 3 minutes to actually start the first step (K8s node auto scaling)
o

owen

04/18/2023, 9:57 PM
ah thanks for the fast response! that does seem to align roughly with the theory that the in-progress run was not being handled properly (as it seems like the second materialization of
predictions
was kicked off right after
inference_data
completed in the original run, rather than waiting for the original run to complete)
d

Daniel Gafni

04/18/2023, 9:58 PM
So by the time the second run got scheduled predictions in the first run didn’t start yet
o

owen

04/18/2023, 9:58 PM
just to confirm, which version of dagster are you on?
d

Daniel Gafni

04/18/2023, 9:58 PM
1.2.7
👍 1
o

owen

04/18/2023, 9:59 PM
and is the timestamp of the materialization of
inference_data
in the first run ~2:14AM?
d

Daniel Gafni

04/18/2023, 10:01 PM
image.png
Meanwhile second run
o

owen

04/18/2023, 10:02 PM
great thank you -- this gives me a lot to work with 🙏. I'll look into it and get back to you
d

Daniel Gafni

04/18/2023, 10:02 PM
🙂 thank you too!
o

owen

04/18/2023, 10:19 PM
ah, I see what's going on here. it's an easy fix but I think it'll need to wait until next week as this week's release is currently being rolled out 😞
d

Daniel Gafni

04/18/2023, 10:23 PM
Ok, thanks fine. Thank you for the fix!
Seems like this sensor is a really complex one
o

owen

04/18/2023, 10:26 PM
that's accurate haha -- the main driving factor behind the complexity is performance (simple solutions end up killing the database or choking on large asset graphs blob smile sweat)
3 Views