Random note about accept-encoding that might have been corrupted by my memory: at some point in the past wikipedia was blocking Googlebot due to it hammering their servers.  Some debugging revealed that Googlebot&#x27;s accept-encoding header was different than browsers and so it was missing the wikipedia cache.  (Wikipedia heavily relies (relied?) upon a caching layer because their pages can take like two full seconds of CPU time to render otherwise.)<p>The kinda awful fix was to add support for the compression variant nobody uses (deflate?) to Googlebot so that its accept-encoding header matched the common one.<p>It would be ironic if adding this header to Firefox wrecked their page load performance for wikipedia!