title: How to rank a million images with a crowdsourced sort
body: <p>I'd like to rank a collection of landscape images by making a game whereby site visitors can rate them, in order to find out which images people find the  most appealing.</p>
  
  <p>What would be a good method of doing that?</p>
  
  <ul>
  <li><strong>Hot-or-Not style</strong>? I.e. show a single image, ask the user to rank it from 1-10. As I see it, this allows me to average the scores, and I would just need to ensure that I get an even distribution of votes across all the images. Fairly simple to implement.</li>
  <li><strong>Pick A-or-B</strong>? I.e. show two images, ask user to pick the better one. This is appealing as there is no numerical ranking, it's just a comparison. But how would I implement it? My first thought was to do it as a quicksort, with the comparison operations being provided by humans, and once completed, simply repeat the sort ad-infinitum.</li>
  </ul>
  
  <p>How would <em>you</em> do it?</p>
  
  <p><em>If you need numbers, I'm talking about one million images, on a site with 20,000 daily visits. I'd imagine a small proportion might play the game, for the sake of argument, lets say I can generate 2,000 human sort operations a day! It's a non-profit website, and the terminally curious will find it through my profile :)</em></p>