The weirdest bug I've ever encountered
Once upon a time, a local computer club chapter I was in bought a bunch of 1Gb PCI&#x2F;PCI-X Intel e1000 NICs to upgrade our systems, and a shiny used GbE switch.<p>We installed them, and a curious thing happened - on _a few_ systems, eventually the NIC would effectively hang, flooding dmesg with &quot;Rx Unit Hang&quot; or &quot;Tx Unit Hang&quot;, and since the systems were NFS+LDAP based, people without local account access were SOL, and a reboot was required to get the NIC back into a good state.<p>When swapping NICs with a system (also using an e1000) where this _never_ happened, the problem did not follow the NIC, it stayed on the original system - same statement if you duplicated the OS install from the working one onto the broken one.<p>So we reached out to Intel&#x27;s Linux support (which, at the time, took posts on their sourceforge page), and they replied basically &quot;hey, yeah, we&#x27;ve heard a few reports of that, but we haven&#x27;t been able to repro it on our testbeds yet&quot;.<p>&quot;We have two identical systems which both do it, would you like to borrow one?&quot;<p>Their reply was a shipping label, so we shipped it to them.<p>A little while later, I got a phone call, and the first words out of the engineer&#x27;s mouth were &quot;Do you have the BIOS that was on this motherboard?&quot;<p>&quot;...what do you mean, was?&quot;<p>Apparently they plugged in a PCI bus analyzer that made assumptions that were true on Intel x86 and not AMD x86, and we had shipped them a Sempron, so comedy ensued.<p>After a number of rabbit holes, the root cause came back - our affected systems were _so_ cheap and terrible, _they lied about bus DMAs succeeding sometimes_, so sometimes the &quot;hey I put&#x2F;cleared this buffer on the NIC&quot; messages were getting lost but not reporting they were lost, and eventually, you run out of buffers...<p>They couldn&#x27;t reproduce it because none of their testbeds were bargain-basement AMD boards. XD<p>They asked us to keep the system for their test set, and shipped us a brand spanking new Core 2 Quad with all the bells and whistles.