Modeling Clutter Perception using Parametric Proto-object Partitioning
Visual clutter, the perception of an image as being crowded and disordered, affects aspects of our lives ranging from object detection to aesthetics, yet relatively little effort has been made to model this important and ubiquitous percept. Our approach models clutter as the number of proto-objects segmented from an image, with proto-objects defined as groupings of superpixels that are similar in intensity, color, and gradient orientation features. We introduce a novel parametric method of clustering superpixels by modeling mixture of Weibulls on Earth Mover's Distance statistics, then taking the normalized number of proto-objects following partitioning as our estimate of clutter perception. We validated this model using a new 90-image dataset of real world scenes rank ordered by human raters for clutter, and showed that our method not only predicted clutter extremely well (Spearman's ρ = 0.8038, p < 0.001), but also outperformed all existing clutter perception models and even a behavioral object segmentation ground truth. We conclude that the number of proto-objects in an image affects clutter perception more than the number of objects or features.
UPDATED AFTER AUTHOR FEEDBACK AND OTHER REVIEWS: 
      Based on the author rebuttal and the comments of the other 
      reviewers, I still believe the paper is worthy of acceptance. However, I 
      agree with the other reviewers that the paper merits a ``1'' (incremental) 
      rather than a ``2'' (major novelty) in the impact score. ORIGINAL 
      REVIEW: Summary: This paper proposes an image-based model for 
      visual clutter perception (``a crowded, disorderly state''). For a given 
      image, the model begins by applying an existing superpixel clustering then 
      computing the intensity, colour and orientation histograms of pixels 
      within each superpixel. Boundaries between adjacent superpixels are then 
      retained or merged to create ``proto-objects''. The novel merging 
      algorithm acts on the Earth Movers Distance (EMD), a measure of the 
      similarity between two histograms. The distribution of histogram distances 
      in each image for each image feature is modeled as a mixture of two 
      Weibull distributions. The crossover point between the two distributions 
      (or a fixed cumulative percentile if a single distribution is preferred by 
      model selection) is used as the threshold point for merging: an edge is 
      labelled ``similar'', and the superpixels merged, if the pair of 
      superpixels exceed the threshold point for all three features. The clutter 
      value for each image is the ratio of the final number of proto-objects to 
      the initial number of superpixels (i.e. 0 = no proto-objects, not 
      cluttered; 1 = all superpixels are proto-objects). The model is 
      validated by comparing to human clutter rankings of a subset of an 
      existing image database. Human observers rank images from least to most 
      cluttered, then the median ranking for each image is used as the ground 
      truth for clutter perception. The new model correlates more highly with 
      human rankings of clutter than a number of previous clutter perception and 
      image segmentation models (including human object segmentation from a 
      previous study). Strengths: In my opinion, scene clutter (a 
      ``crowded, disorderly state'') is not the most interesting quality of 
      perception, but a workable definition of ``set size'' in complex scenes is 
      a very important problem. I think this paper is the precursor to an 
      important contribution: namely, whether this clutter metric relates to 
      performance in behavioural tasks such as search. The method for clustering 
      seems to do a good job and to my knowledge is a sufficiently different 
      approach to previous methods as to offer a novel contribution. The 
      behavioral ranking task is neat and the dataset will be of interest to the 
      community for future model testing. The model correlates admirably with 
      clutter rankings. I believe the paper to be a good contribution to the 
      NIPS community. Weaknesses: It is not clear from the 
      manuscript how strongly the success of the clutter metric depends on the 
      type and parameters of the superpixel pre-processing. I assume this is 
      done to make computation more tractable, but more detail should be 
      provided in the manuscript as to why this does / does not matter. 
      The paper makes the theoretical (based on previous work) and 
      empirical (based on three histograms) claim that the similarity distances 
      follow Weibull distributions. I am not convinced beyond doubt that this is 
      necessarily true. For example, I wonder whether the histograms in Figure 2 
      would be equally well fit by other mixture models using e.g. Gamma 
      distributions. However, I don't think this distinction is important for 
      the applicability of the method. Clarity: The manuscript is 
      generally clearly written, but I believe some re-ordering of 
      methodological details could save the reader some effort. For example, we 
      are told in Section 2.1 that superpixels are extracted, but not until line 
      336 is the actual method referenced. Line 149: ``However, there is no 
      evidence showing that similarity distances follow symmetrical or normal 
      distributions. Therefore, we argue that the similarity distances xf 
      computed by EMD follow a Weibull distribution, which is a distribution of 
      the Exponential family that is skewed in shape.'' The ``therefore'' 
      implies that the Weibull distribution is a logical given, but it's not at 
      all clear from this how a Weibull distribution was selected. The choice is 
      justified more in subsequent paragraphs but the paper would be more clear 
      if these paragraphs were rearranged: first tell us why you selected 
      Weibull then tell us ``for these reasons we argue... Weibull''. 
      Minor points / suggestions: -- for the initial MLE guess (line 
      190): I understand that this would be set separately for every image. Is 
      this correct? -- Line 269: ``making p and epsilon the only model 
      parameters in our framework'': Should this read ``tau'', not ``p''? If 
      not, what does ``p'' refer to here? -- is it correct that each 
      image will contain a potentially different threshold point, depending on 
      the fitting of the mixture model to the distance histograms from that 
      image? -- section 4.1 is unclear to me: does ``image features'' 
      here refer to the similarity distance metrics in each of intensity, color 
      and orientation, or the intensity, color and orientation histograms 
      themselves? In the latter case, is it right to say that a circular 
      statistic such as orientation is ``upper bounded''? -- lines 
      347--350: ``visual clutter perception may ignore feature dissimilarity on 
      the order of 14% when deciding whether two adjacent regions are similar 
      and should be merged.'' Given that the largest value of the epsilon 
      parameter tested was 15%, I would want to see some dramatic drop of 
      correlation within that 14--15% range to believe this more. A plot of the 
      average correlations against epsilon (with five curves, one for each tau) 
      would be informative here. -- it is interesting that human object 
      count is such a poor predictor of clutter. Perhaps the authors could 
      provide example images from the database to show where clutter rankings 
      and object counts diverge. -- is it meaningful to ask why the 
      correlations of the clutter model with the median ranking is higher than 
      the inter-observer agreement? On line 309 you present the average 
      correlation between pairs of participants. What is the correlation of each 
      participant with the average of all other participants? That is, does the 
      clutter model predict the average ranking better than any individual 
      human? -- the reader is often provided with an average rank order 
      correlation and a p-value (presumably telling us that this correlation is 
      significantly different to zero). I would find it more informative to be 
      given the mean correlation and also the range of observed correlations. 
      -- First sentence of section 2.4 has an incorrect figure 
      reference; should be Fig 2d? -- Line 399 missing capital letter to 
      start sentence. -- Line 401 "our work suggests a mean" should be 
      "means"