Just ran into a new error this morning on k8s Dagster Using

Question

Just ran into a new error this morning on k8s Dagster Using the default k8s postgres DB and the s3 io manager Just wiped them both out the postgres volume and the S3 prefix and deployed fresh still seeing it the first time I trigger a run It seems like it s an issue with Dagit talking to the Dagster user deployment service but I m not entirely sure Error deets will be in thread salient message ```Message Attempted to deserialize class EventLogEntry which is not in the whitelist This error can occur due to version skew verify processes are running expected versions ```

Michel Rouly · Answer

```Message Event logs invalid for run id e2a63b3d db61 4cfd 8dba 69f20e4e5eed Path Locations Stack Trace File usr local lib python3 7 site packages rx core observablebase py line 67 in set disposable subscriber = self subscribe core auto detach observer File usr local lib python3 7 site packages rx core anonymousobservable py line 20 in subscribe core return self subscribe observer File usr local lib python3 7 site packages dagster graphql implementation pipeline run storage py line 11 in call events = self instance logs after self run id self after cursor File usr local lib python3 7 site packages dagster core instance init py line 1023 in logs after return self event storage get logs for run run id cursor=cursor of type=of type File usr local lib python3 7 site packages dagster core storage event log sql event log py line 200 in get logs for run events by id = self get logs for run by log id run id cursor of type File usr local lib python3 7 site packages dagster core storage event log sql event log py line 179 in get logs for run by log id raise DagsterEventLogInvalidForRun run id=run id from err The above exception was the direct cause of the following exception Message Attempted to deserialize class EventLogEntry which is not in the whitelist This error can occur due to version skew verify processes are running expected versions Descent path lt root dict gt Stack Trace File usr local lib python3 7 site packages dagster core storage event log sql event log py line 176 in get logs for run by log id deserialize json to dagster namedtuple json str event EventRecord File usr local lib python3 7 site packages dagster serdes serdes py line 265 in deserialize json to dagster namedtuple check str param json str json str whitelist map= WHITELIST MAP File usr local lib python3 7 site packages dagster serdes serdes py line 277 in deserialize json return unpack value value whitelist map=whitelist map descent path= root value File usr local lib python3 7 site packages dagster serdes serdes py line 308 in unpack value f Attempted to deserialize class klass name which is not in the whitelist ```

Michel Rouly · Answer

Dagit and user deployment are on `0 11 14`

Michel Rouly · Answer

going to try rolling forward to 0 11 15 but I wish I knew what was causing this

daniel · Answer

Hi the most likely cause here is that your dagster package is on an earlier version than 0 11 14 reinstalling your dagit and making sure to do an upgrade of the dagster package should fix this We actually just shipped a change in 0 11 15 that should make this version skew much harder to happen dagit 0 11 15 is pinned to dagster 0 11 15 so updating would also be an option

Michel Rouly · Answer

Nice Sounds like rolling forward is the plan

Michel Rouly · Answer

Forgot to update rolling forward to 0 11 15 totally addressed the problem