So one architectural issue we've struggled with (and that it seems from our research the whole community has issues with) is how to garbage collect ephemeral Pods (specifically, Jobs). It seems like folks must be using CRDs/custom controllers for this kind of thing -- there is the TTL Controller for Finished Resources and we have set

ttlSecondsAfterFinished

in our Helm chart to try to take advantage of this, but it's still alpha in k8s v1.12