19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Recently, I've been seeing this error on job retri...
# ask-communityc
Charlie Bini
04/03/2023, 4:23 PMRecently, I've been seeing this error on job retries. Any idea what the problem might be?
Copy code
An exception was thrown during execution that is likely a framework error, rather than an error in user code.
dagster._check.CheckError: Invariant failed. Description: __ASSET_JOB_1 has no op named kippcamden__powerschool__assignmentscore.Charlie Bini
04/03/2023, 4:24 PMit's from a job launched by a
build_asset_reconciliation_sensors
sandy
04/03/2023, 7:57 PMWhen you say "on job retries", do you mean that you're manually retrying a job that was launched by the sensor?
c
Charlie Bini
04/03/2023, 7:57 PMauto-retry
s
sandy
04/03/2023, 7:57 PMInteresting - and are you reloading your code location in between by any chance?
c
Charlie Bini
04/03/2023, 7:58 PMso job launched by the sensor failed, and upon all subsequent retries, it fails with that
Charlie Bini
04/03/2023, 7:58 PMI'm not
s
sandy
04/04/2023, 12:02 AMsomeone else reported a similar issue today
a couple more followup questions:
• do you have multiple code locations with assets in them?
• if you go to the page with your sensors and click on the link to the relevant asset job, does it contain the op/asset you expect?
sandy
04/04/2023, 3:40 PMand one more:
• does this happen every time there's a job retry? is it a rare occurrence?
c
Charlie Bini
04/04/2023, 3:43 PMhey @sandy! yes to question 1: we do have multiple code locations with assets. I'm unsure of the other 2 but I'll look into them today
s
sandy
04/04/2023, 3:44 PMbtw here's an issue for tracking: https://github.com/dagster-io/dagster/issues/13360
c
Charlie Bini
04/04/2023, 5:53 PMok so to Q3: yes, almost all retries created by the asset reconciliation sensor fail
Charlie Bini
04/04/2023, 5:53 PMthe last successful retry I had with that was on March 29
Charlie Bini
04/04/2023, 5:54 PMall of the failed retries have a "Failed to start" status in the Runs view
Charlie Bini
04/04/2023, 5:55 PMlooking into the asset jobs question now
Charlie Bini
04/04/2023, 6:07 PMthis seems odd: there are ~50
__ASSET_JOBCharlie Bini
04/04/2023, 6:07 PMit looks like all except the last column are not partitioned assets
Charlie Bini
04/04/2023, 6:09 PMonly the dbt assets have been added to the sensor, but the asset list on the job includes their upstream asset
Charlie Bini
04/04/2023, 6:10 PMthese assets aren't attached to any sensor or dbt model, yet they appear on the sensor's jobs
s
sandy
04/04/2023, 6:40 PMany chance you'd have some time this afternoon or tomorrow to jump on a call to debug this?
c
Charlie Bini
04/04/2023, 6:48 PMthis afternoon could be tricky but definitely tomorrow
s
sandy
04/04/2023, 8:10 PMWould 1 pm pacific work for you tomorrow? I should be free any time in the afternoon other than 2-2:30 pm
c
Charlie Bini
04/04/2023, 8:11 PMyeah 1 PT would be perfect
s
sandy
04/04/2023, 8:16 PMgreat - what's a good email address to send a calendar invite to?
c
Charlie Bini
04/04/2023, 8:16 PMCharlie Bini
04/04/2023, 9:25 PMomg I don't know my own email sorry
Charlie Bini
04/04/2023, 9:25 PMCharlie Bini
04/04/2023, 9:25 PMwe've changed our domain a few times since I've been here
s
sandy
04/04/2023, 10:26 PMah added that email address
j
johann
04/07/2023, 8:36 PM@Charlie Bini we’ve landed a fix that will go out in the release next week
9 Views