Unsolved mystery from previous job: sysvinit 2.86 failing to reap dead processes, linux-2.6.26, MIPS64
This was a problem I never solved at a previous startup company that no longer exists. QA were running some tests on the MIPS hardware by blasting commands at it with the `rsh` utility.

Sometimes, thousands of the `bash` processes would turn into zombies, not collected by `init`. But if you had the system in this situation (hard to reproduce!), then a `kill -HUP 1` to kick the init daemon (or `telinit q`) would cause it to reap those processes instantly.

I and at least one other colleague pored over the kernel source code, and the `init 2.86` source code (with some patches applied to it from various sources). Never found the reason how it could happen. I could not come up with a hypothesis about how init could cause it; I could not convince myself that it has any race condition w.r.t. responding to the `SIGCHLD` signal and reaping the children. So then the line of investigation was: init isn't losing the wakeup; the kernel is bungling it. After a lot of code reading and forming various hypotheses (only to disprove them), the search dried up, and there were higher priority problems to investigate. 

It's been a few years since then.

Has anyone run into this and solved it?

I just had a glance through more recent `sysvinit` change logs, but nothing stands out. I chased the reference to a patch against `2.88` from F. Viard, (Savannah bug 32304) but I don't think that's it.
Unsolved mystery from previous job: sysvinit 2.86 failing to reap dead processes, linux-2.6.26, MIPS64
===

This was a problem I never solved at a previous startup company that no longer exists. QA were running some tests on the MIPS hardware by blasting commands at it with the `rsh` utility.

Sometimes, thousands of the `bash` processes would turn into zombies, not collected by `init`. But if you had the system in this situation (hard to reproduce!), then a `kill -HUP 1` to kick the init daemon (or `telinit q`) would cause it to reap those processes instantly.

I and at least one other colleague pored over the kernel source code, and the `init 2.86` source code (with some patches applied to it from various sources). Never found the reason how it could happen. I could not come up with a hypothesis about how init could cause it; I could not convince myself that it has any race condition w.r.t. responding to the `SIGCHLD` signal and reaping the children. So then the line of investigation was: init isn't losing the wakeup; the kernel is bungling it. After a lot of code reading and forming various hypotheses (only to disprove them), the search dried up, and there were higher priority problems to investigate. 

It's been a few years since then.

Has anyone run into this and solved it?

I just had a glance through more recent `sysvinit` change logs, but nothing stands out. I chased the reference to a patch against `2.88` from F. Viard, (Savannah bug 32304) but I don't think that's it.