Learning the Local Statistics of Optical Flow
Motivated by recent progress in natural image statistics, we use newly available datasets with ground truth optical flow to learn the local statistics of optical flow and compare the learned models to prior models assumed by computer vision researchers. We find that a Gaussian mixture model (GMM) with 64 components provides a significantly better model for local flow statistics when compared to commonly used models. We investigate the source of the GMM's success and show it is related to an explicit representation of flow boundaries. We also learn a model that jointly models the local intensity pattern and the local optical flow. In accordance with the assumptions often made in computer vision, the model learns that flow boundaries are more likely at intensity boundaries. However, when evaluated on a large dataset, this dependency is very weak and the benefit of conditioning flow estimation on the local intensity pattern is marginal.
The paper proposes using recent large-scale optical 
      flow datasets to learn priors on optical flow. Two types of priors are 
      learned: a prior on flow vectors, and a joint prior on intensity and flow. 
      GMMs are used for both priors. It is shown that these rich data-driven 
      priors outperform previous ad-hoc priors on flow restoration tasks. On the 
      other hand, the advantages of these priors for actual optical flow 
      estimation are not demonstrated. Overall, this is a nice paper, 
      albeit a bit narrow. I am positive on it and support acceptance, in part 
      because I work in the area and find some of the observations made in this 
      paper to be useful. Pros: - Natural idea, it's good that 
      somebody did it, nice to know what the results are. - It's nice that 
      the likelihood estimation results translate across datasets. - The 
      observation that flow boundaries are not strongly correlated with 
      intensity boundaries is very nice and is the highlight of the paper. It's 
      not completely clear that the evidence in the paper actually supports 
      making this claim broadly and strongly, but the evidence is certainly 
      suggestive. This observation is a stimulating addition to the literature. 
      I would like to see it published. Cons: - Benefits to 
      actual optical flow estimation are not demonstrated, and are potentially 
      marginal. The paper's contribution seems to be mostly conceptual rather 
      than practical. - With regards to the intriguing results described in 
      line 373 and onwards: could this be due to the suboptimal learning 
      algorithm? Can we really draw general conclusions from these experiments, 
      or should the conclusions be qualified a bit due to the limitations of the 
      learning procedure? Or the data? Specific/minor comments: 
      - lines 73, 74: "(figure 1b)", "city [5]" - line 82: "GMM's", 
      "show that it is" - equation (1): fix parentheses and brackets in 
      multiple places - line 355: "of of" -> "of" - lines 300-323 and 
      figure 6: I found it hard to follow the text and to see precisely the 
      phenomena that are being described in Figure 6. The figure should be 
      annotated much better. The phenomena should be visually apparent. I just 
      don't see some of the things that are being described in the figure. - 
      lines 357-358: what intensity weighting did you use for this model and 
      why? - line 424: "assumed priors"