The problem is how to debug sporadic production problems.  It doesn&#x27;t matter how good your tooling is to debug a problem in a dev environment if you have no idea how to reproduce it there.<p>You need some level of decent logging in the production environment (with optional extra logging that can be turned on) to capture WTF went wrong.  THEN you can try to reproduce it.  When a logged system goes boom, you need it to come out of production and remain around until those logs are saved somewhere.<p>It is old, but <a href="http:&#x2F;&#x2F;lkml.iu.edu&#x2F;hypermail&#x2F;linux&#x2F;kernel&#x2F;0009.1&#x2F;1307.html" rel="nofollow">http:&#x2F;&#x2F;lkml.iu.edu&#x2F;hypermail&#x2F;linux&#x2F;kernel&#x2F;0009.1&#x2F;1307.html</a> is an interesting data point about IBM&#x27;s experience.  They implemented exactly the kind of logging that I advocate in all of their systems until OS&#x2F;2.  And the fact that they didn&#x27;t have it in OS&#x2F;2 was a big problem.