I'm getting an intermittant error when dagit is st...
# announcements
I'm getting an intermittant error when dagit is starting:
Object IPCErrorMessage(serializable_error_info=SerializableErrorInfo(message="dagster.check.CheckError: Invariant failed. Description: Can only serialize whitelisted namedtuples, received tuple ('2018-01-01', None)
but I'm having a lot of trouble finding the source of the Tuple. Thoughts on where this would be coming from?
Full output^^
hmm any chance that date is hardcoded anywhere? A rogue
in yaml or python could be turning something in to a tuple on accident
There are a few places where the date is hardcoded, but I've systematically changed all of them and the date in the error hasn't changed. I'm afraid it's something to do with our kubernetes setup
umm double check that you are using the image&tag you expect and are not on a stale one by accident?
image pull policy may be a thing too, if you are re-pushing the same image tag
It had to do with docker image reuse: forcing a new image seems to have fixed it
👍 1