Tell Me More, Nginx (2017)
The company that I recently joined uses honeycomb.io, versus ELK at my previous job. Maybe I haven't played with it enough, or maybe we have it badly configured, but I find it a huge step backwards. As far as I know, there's no full text search and the types of reports and aggregation you can do are extremely primitive. The UI is also barely usable..huge horizontal scrolling, can't click on a specific value to apply that sub-filter. The list goes on and on. I wish devops team would focus on these fundamentals (proper centralized logging) before playing with all the "cool" and trendy toys.<p>As for logging, Our `log_format` directive looks like:<p><pre><code>  log_format stats '$time_iso8601\t$host\t$request_method\t$request_uri\t$request_length\t$status\t$upstream_cache_status\t$bytes_sent\t$request_time\t$upstream_response_time\t$upstream_http_x_route\t$client_id';
</code></pre>
This captures most of the generic data you'd associate with a request (time, host, method, uri, length) as well as the generic response data you'd care about (status, length, time to reply).  Note that `$request_time` measures the full time to first byte until time to last byte that nginx spends serving the request, vs $upstream_response_time which is more specific to the upstream's response time.<p>If you're using nginx as a cache, $upstream_cache_status tells you the cache hit/miss status.<p>A list of all variables is available at <a href="http://nginx.org/en/docs/varindex.html" rel="nofollow">http://nginx.org/en/docs/varindex.html</a><p>All of our services can set an "x-route" header, which helps canonicalize URIs into something more meaningful. It's up the the service to decide what to do..but /v1/users/ID could be called "users:show". A more complex route might use a different name based on arguments. For example, depending on the parameters we expect some hits to /v1/reports to be very fast, and some to be very slow, so we'll set the name to "reports:list:fast" or "reports:list:slow" so that we can get more accurate statistics.<p>Finally, we use OpenRestry to do some initial global authentication (along with some other stuff). This is where $client_id comes from. All requests go through something like:<p>location / {
  set $client_id ''
  set $upstream ''
  access_by_lua_block {
    require('execute')()
  }
  proxy_pass <a href="http://$upstream;" rel="nofollow">http://$upstream;</a>
}<p>A piece of that execute code will possibly set $client_id to something which will then get logged.