https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Charlie Bini

04/03/2023, 4:23 PM
Recently, I've been seeing this error on job retries. Any idea what the problem might be?
Copy code
An exception was thrown during execution that is likely a framework error, rather than an error in user code.

dagster._check.CheckError: Invariant failed. Description: __ASSET_JOB_1 has no op named kippcamden__powerschool__assignmentscore.
it's from a job launched by a
build_asset_reconciliation_sensor
sensor
s

sandy

04/03/2023, 7:57 PM
When you say "on job retries", do you mean that you're manually retrying a job that was launched by the sensor?
c

Charlie Bini

04/03/2023, 7:57 PM
auto-retry
s

sandy

04/03/2023, 7:57 PM
Interesting - and are you reloading your code location in between by any chance?
c

Charlie Bini

04/03/2023, 7:58 PM
so job launched by the sensor failed, and upon all subsequent retries, it fails with that
I'm not
s

sandy

04/04/2023, 12:02 AM
someone else reported a similar issue today a couple more followup questions: • do you have multiple code locations with assets in them? • if you go to the page with your sensors and click on the link to the relevant asset job, does it contain the op/asset you expect?
and one more: • does this happen every time there's a job retry? is it a rare occurrence?
c

Charlie Bini

04/04/2023, 3:43 PM
hey @sandy! yes to question 1: we do have multiple code locations with assets. I'm unsure of the other 2 but I'll look into them today
s

sandy

04/04/2023, 3:44 PM
btw here's an issue for tracking: https://github.com/dagster-io/dagster/issues/13360
c

Charlie Bini

04/04/2023, 5:53 PM
ok so to Q3: yes, almost all retries created by the asset reconciliation sensor fail
the last successful retry I had with that was on March 29
all of the failed retries have a "Failed to start" status in the Runs view
looking into the asset jobs question now
this seems odd: there are ~50
__ASSET_JOB
listed under the sensor, and they all show roughly 47 assets, some of which I didn't add to the sensor
it looks like all except the last column are not partitioned assets
only the dbt assets have been added to the sensor, but the asset list on the job includes their upstream asset
these assets aren't attached to any sensor or dbt model, yet they appear on the sensor's jobs
s

sandy

04/04/2023, 6:40 PM
any chance you'd have some time this afternoon or tomorrow to jump on a call to debug this?
c

Charlie Bini

04/04/2023, 6:48 PM
this afternoon could be tricky but definitely tomorrow
s

sandy

04/04/2023, 8:10 PM
Would 1 pm pacific work for you tomorrow? I should be free any time in the afternoon other than 2-2:30 pm
c

Charlie Bini

04/04/2023, 8:11 PM
yeah 1 PT would be perfect
s

sandy

04/04/2023, 8:16 PM
great - what's a good email address to send a calendar invite to?
c

Charlie Bini

04/04/2023, 8:16 PM
omg I don't know my own email sorry
we've changed our domain a few times since I've been here
s

sandy

04/04/2023, 10:26 PM
ah added that email address
j

johann

04/07/2023, 8:36 PM
@Charlie Bini we’ve landed a fix that will go out in the release next week
9 Views