<p dir="auto">A max number of failures or failure backoff policy for Jobs would be useful.</p>
<p dir="auto">Imagine you have an ETL job in production that's failing due to some pathological input.<br>
By the time you spot it, it's already been rescheduled thousands of times. As an example, I had 20 broken jobs that kept getting rescheduled; killing them took forever -- and crashed the Kubernetes dashboard in the process.</p>
<p dir="auto">Today, "restartPolicy" is not respected by Jobs because the goal is to achieve successful completion. (Strangely, "restartPolicy: Never" is still valid YAML.) This means failed jobs keep getting rescheduled. When you go to delete them, you have to delete all the pods they've been scheduled on. Deletes are rate limited, and in v1.3+, the verbose "you're being throttled" messages are hidden from you when you run the <code class="notranslate">kubectl</code> command to delete a job. So it just looks like it's taking forever! This is not a pleasant UX if you have a runaway job in prod or if you're testing out a new job and it takes minutes to clean up after a broken test.</p>
<p dir="auto">What are your thoughts on specifying a maximum number of failures/retries or adding a failure backoff restart policy?</p>
<p dir="auto">(cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/thockin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/thockin">@thockin</a> , who <a href="https://twitter.com/zazius/status/758816688171126784" rel="nofollow">suggested I file this feature request</a>)</p>
<hr>
<p dir="auto">Ninja edit: <a href="http://stackoverflow.com/questions/35141607/is-there-a-max-retries-for-kubernetes-jobs" rel="nofollow">per this SO thread</a>, the throttling issue is avoidable by using the OnFailure restart policy to keep rescheduling to the same pods, rather than to new pods -- i.e. this prevents the explosion of the number of pods. And deadlines can help weed out failures after a certain amount of time.</p>
<p dir="auto">However, suppose my ETL job takes an hour to run properly but may fail within seconds if the input data is bad. I'd rather specify a maximum number of retries than a high deadline.</p>
<p dir="auto">Today <code class="notranslate">kubectl exec sh</code> depends on a shell being present in each container image.  I think we can do better.</p>
<p dir="auto">What if we had something like <code class="notranslate">kubectl sh</code> which would run a statically linked busybox (or something like that) that was provided by the host node and made that enter the cgroups and namespaces and chroot of the container?  We already have a short list of features that only work if the host node has some piece of software installed, this would be the same.</p>