Hidden latency in the Linux network stack
Stray thoughts:<p>1) Any chance this bug would have manifested as &quot;connection reset&quot; errors when accessing HN?   I exchanged email with Dan a couple months ago trying to figure out why about 10% of my requests were failing, but we never figured out the root cause before (after some weeks) the problem went away.<p>2) As others have pointed out, doubling the number of hash buckets seems like a bandaid.  But other than the scolding comment, is there any reason not to go to an appropriately sized hash?  If you know in advance that you are going to have 16K addresses (ie, not the use case the original code anticipated), it would seem beneficial to choose a data structure that has a fighting chance of providing good performance.<p>3) This seems like a wonderful argument for _against_ running your high performance DNS server on the same machine as your other services.  Would containerization have helped here, possibly with each OS pinned to a set of cores? Is the cost of splitting it off onto a separate physical machine prohibitive?  At the optimization level you are aiming for, &quot;process isolation&quot; seems like a pretty leaky abstraction.<p>4) Going farther down the path of reducing jitter, have you considered running a tickless (NOHZ_FULL) kernel?  Perhaps you are already doing it, but quieting down the cores running your important services can make a significant difference.  I&#x27;ve been exploring this direction, and have found it rewarding. Info on one way to measure is here: <a href="https:&#x2F;&#x2F;kernel.googlesource.com&#x2F;pub&#x2F;scm&#x2F;linux&#x2F;kernel&#x2F;git&#x2F;frederic&#x2F;dynticks-testing&#x2F;+&#x2F;master" rel="nofollow">https:&#x2F;&#x2F;kernel.googlesource.com&#x2F;pub&#x2F;scm&#x2F;linux&#x2F;kernel&#x2F;git&#x2F;fre...</a>