Is this the expected behavior on GKE when `max retries` is g

Question

Is this the expected behavior on GKE when `max retries` is gt 0 When jobs fail it starts a new pod resuming the previous job at the point of failure However I m using the default IO Manager `fs io manager` I believe so the retry fails with `FileNotFoundError Errno 2 No such file or directory opt dagster dagster home storage 030b6a20 fea9 481e 8ac2 c2fcca035741 query data get count storedgrades q 0 table ` Is using the `gcs pickle io manager` required for using the retry feature

Charlie Bini · Answer

If so is there a way to set the deployment to use `gcs pickle io manager` as the default or does it need to be added to every job

daniel · Answer

yeah to be able to pick off where it left off it needs to be able to pull outputs from the failed steps so either mounting a shared filesystem somehow or using an IO manager that actually persists results like the gcs io manager is the way to go

daniel · Answer

For the second part We re actually thinking right now about ways to set resources at the repo level which would address this but for now I think it would need to be configured on every job yeah

Charlie Bini · Answer

gotcha thanks for clearing that up

Charlie Bini · Answer

< marcos> FYI point up