title: Comparing histograms for jpeg image analysis using raster stack
body: <p>I have thousands of webcam images in jpeg format which I want to analyze for cloud cover. Images with no cloud will be a mountain and will be relatively dark while if there is cloud the histogram is skewed to bright levels. The aim is to classify images into cloud or no cloud and then analyze the frequency of cloud by time of day, season, etc.</p>
  <p>First the jpegs are imported using <code>raster</code>. I intend to compile the images into a <a href="https://artax.karlin.mff.cuni.cz/r-help/library/raster/html/stack.html" rel="nofollow noreferrer">RasterStack</a> and specify a window within each image for the analysis, e.g.</p>
  <pre><code>rasterlist &lt;-  list.files('test', full.names=TRUE) # list of files to make rasterstack
  test_stack &lt;- stack(rasterlist) # stack raster layers
  
  #define the crop extent
  cropbox &lt;-c(200,1280,500,900) # (`xmin`, `xmax`, `ymin` , `ymax`)
  test_cr &lt;- crop(test, cropbox) #crop the raster
  </code></pre>
  <p>I am looking for a robust dissimilarity measure to compare all of the images in the stack to a reference image using histograms. The more threads I read the more metrics I come across as suggestions! <a href="https://stackoverflow.com/questions/6499491/comparing-two-histograms">This SO thread</a> provides a good starting point.</p>
  <p>The jpeg pixels have 256 values when unbinned. My understanding is the Kolmogorov-Smirnov distance is an appropriate measure in this case, or chi-squared distance if the data are binned (which may be necessary to improve computation speed).</p>
  <p>Several discussions on this topic recommend Wasserstein or Earth Movers Distance (EMD) but when I try using this (package <a href="https://cran.r-project.org/web/packages/transport/index.html" rel="nofollow noreferrer">transport</a>) on a pair of test images (each 1080x250 pixels) I get an error msg (apparently my 16GB of RAM is not nearly enough!):</p>
  <pre><code># first need to convert raster to a matrix then to object type pgrid
  testmat &lt;- as.matrix(test_cr2)
  testmat2 &lt;- as.matrix(test_cr3)
  p1 &lt;- pgrid(testmat)
  p2 &lt;- pgrid(testmat2)
  wasserstein(p1,p2,p=1) # calculate EMD
  
  Error: cannot allocate vector of size 271.6 Gb
  </code></pre>
  <p>I have had more success with <a href="https://www.rdocumentation.org/packages/HistogramTools/versions/0.3.2/topics/HistogramDistance" rel="nofollow noreferrer">HistogramDistance</a> from the HistogramTools package, e.g.</p>
  <pre><code># HistogramDistance requires 2 histogram objects with same bucket boundaries
  h1 &lt;- hist(test_cr2, breaks=seq(0,256,by=8))
  h2 &lt;- hist(test_cr3, breaks=seq(0,256,by=8))
  minkowski.dist(h1, h2, 1) #  1 = manhattan dist
  </code></pre>
  <p>But I can find little information about the relative merits of the 4 different similarity metrics that HistogramDistance uses:</p>
  <blockquote>
  <p>The minkowski.dist function computes the Minkowski distance of order p between two histograms. p=1 is the Manhattan distance and p=2 is the Euclidean distance.</p>
  <p>The intersect.dist function computes the intersection distance of two histograms, as defined in Swain and Ballard 1991, p15. If histograms h1 and h2 do not contain the same total of counts, then this metric will not be symmetric.</p>
  <p>The kl.divergence function computes the Kullback-Leibler divergence between two histograms.</p>
  <p>The jeffrey.divergence function computes the Jeffrey divergence between two histograms.</p>
  </blockquote>
  <p>Is Minkowski distance suited to my purpose? And if so which is better, manhattan or euclidean distance?</p>
  <p>I can find no actual examples of any of these methods being implemented for histogram similarity in R.</p>
  <p>Firstly, I'm after advice on which are the most useful methods that are relatively easy to implement in R for a large dataset (thousands of histograms).</p>
  <p>Secondly, examples of any of these methods implemented in R for histogram comparison. (I can't find any relevant code online).</p>