title: How do i fix my S3/memcache latency problem?
body: <p>We have SERIOUS latency issues SOMETIMES on our server.</p>
  
  <p>We store 3 things of interest in S3 and stuff them into memcache as well.</p>
  
  <ul>
  <li>User avatars averaging ~25k</li>
  <li>text ~1.5k</li>
  <li>xml ~1.5k</li>
  </ul>
  
  <p>we have dedicated 128meg of ram for memcached as of right now...
  as of right now it is riding 74 meg of it</p>
  
  <p>doing some basic math we should easily be able to have around 30,000 text documents (with the xml representation of them) and 1,000 user avatars and still be under our 128meg dedicated to memcache</p>
  
  <p>right now we have ~100 user avatars that could be pulled up at any given time
  we have in the hundred thousands of the text/xml documents but they don't get viewed
  like the avatars do...it's one here, on there type of thing</p>
  
  <p>sometimes during the day the user avatars load super slowly (indicating they have to be loaded from s3) and other times (after they load of course) you can tell they are being served from memcached; same thing with the text documents</p>
  
  <p>we are running merb under apache phusion with REE.
  we are using evan weaver's memcached gem built upon libmemcached-0.25.14 (which I fully understand is not the most up to date lib; this gem requires it)</p>
  
  <p>From what I can see our latency issues are because of S3 which does have serious latency issues (sometimes 500ms for a single avatar). However, it seems that it should not be an issue considering it should be cached all the time. The default expiry on the cache is set to 1 week.</p>
  
  <p>Relevant code is:</p>
  
  <pre><code>@cache = MMCACHE.clone
  begin
    picture = @cache.get("/avatars/#{user.avatar}")
  rescue
    picture = user.picture
    @cache.set("/avatars/#{user.avatar}", picture)
  end
  @cache.quit
  </code></pre>
  
  <p>the cloning/quitting is important since under apache/phusion will have problems sharing the connections when it forks it and if we didn't close out our connections they'd keep building up until we run out of file descriptors.</p>
  
  <p>I'm starting to keep a much closer eye on memcache to see if I can track down my problems but any suggestions?? Should we just get rid of S3??</p>