Monitoring your logs is mostly a tarpit
Request logging should be done by an application gateway; it will do a more honest job of it. In applications, don&#x27;t log anything except:<p>- application startup and shutdown (at INFO level)<p>- occurrences that indicate a bug in the system logging (at ERROR level)<p>- occurrences that indicate a bug in another of your systems (at WARN level)<p>The last two require a lot of education and discipline to stop people from logging rare but valid circumstances, bad input originating outside your software, and &quot;important&quot; events. The answer to &quot;but don&#x27;t we want to know when...&quot; arguments should always be, &quot;If it&#x27;s that important, emit it as a metric or store it in an appropriate datastore.&quot; Anything logged at WARN or ERROR should be something that can be addressed by fixing code.<p>If you stick to this discipline, monitoring logs is valuable and reasonably simple. You can alert on any WARN and ERROR level logging. There&#x27;s nothing like looking at a month of logs and seeing only actionable information and a handful of application lifecycle events. (There&#x27;s also nothing like looking at six months of logs from a stable system and only seeing lifecycle events.)<p>It is hard to stick to this discipline. You need a manager who believes in it and will back it up when engineers want to depart from it, and who will always prioritize fixing violations.<p>Why is it so hard to make engineers follow these rules? Because initially, it makes them feel incredibly anxious. It feels irresponsible to throw away so much &quot;important&quot; information instead of logging it. After working this way for a while, though, they realize that it actually forces a higher level of responsibility. Logging information in noisy logs might feel better than throwing it away, but it&#x27;s not. It creates an illusion of having handled the information responsibly. By taking away the illusory option, you force people to make a real decision between throwing it away and saving it in an actionable way, by emitting it as a metric or storing it in a datastore to be processed.