Multiscale Fields of Patterns
We describe a framework for defining high-order image models that can be used in a variety of applications. The approach involves modeling local patterns in a multiscale representation of an image. Local properties of a coarsened image reflect non-local properties of the original image. In the case of binary images local properties are defined by the binary patterns observed over small neighborhoods around each pixel. With the multiscale representation we capture the frequency of patterns observed at different scales of resolution. This framework leads to expressive priors that depend on a relatively small number of parameters. For inference and learning we use an MCMC method for block sampling with very large blocks. We evaluate the approach with two example applications. One involves contour detection. The other involves binary segmentation.
The authors propose a way to define distributions over high-order image features, using a multiscale representation of binary image features, termed multiscale field of patterns (FoP). The paper discusses two example applications, contour integration and segmentation.- Originality and SignificanceThe idea that multiscale representations can be useful to capture global image properties is at least as old as computer vision. Nonetheless, the paper presents a thorough and original derivation of a prior over higher-order binary features. In particular, if I understand correctly, the nonlinearity (logical OR) used to build the binary image pyramid, and the use of nonlinear filters (binary pattern detectors), allows the model to capture higher-order image properties. For instance, with the right choice of cost (or energy) functions for the binary patterns at different scales, the authors demonstrate that samples from the model trained on contour images can capture long and curved contours. This is an interesting result, and so are the results on contour detection in noisy images and figure/ground segmentation. On the other hand, though, there are some important limitations. First, the authors only consider binary images and white pixel noise, therefore dealing with extremely simplified statistics, compared to natural images. For instance, given that the contour images are taken from the BSD (where both human-labeled contours and the original natural images are provided), it would be interesting to see if the model successfully detects contours also on those images (as opposed to just the binary contour image with white noise added on top). Neither do the authors experiment with white noise at different scales, which would be a slightly stronger test than adding white noise only at the finest level (since the noise averages out at coarser levels). Also, the authors compare the multiscale FoP model to a single scale FoP and to “baseline” models from the literature; but how well does the multiscale FoP compare to state of the art in the applications described?Another major limitation, at present, seems the computational complexity of the algorithm (Line 300 and following: 2 days of training over a cluster with 20 cores for a 200 images training set?), which makes the approach rather impractical. - QualityThe paper is technically sound and of good quality. It presents a simple and clear idea, develops the necessary math in great detail, and shows good results on interesting computer vision problems such as contour detection and figure/ground segmentation.Minor: Line 418: “picked the pairwise discontinuity cost minimizing the per-pixel error rate in the test set”. Doesn’t this produce overfitting?- ClarityThe paper is very well written and provides good coverage of relevant literature. I would suggest to consider also the work of Theis, L., Hosseini, R., & Bethge, M. (2012). Mixtures of conditional Gaussian scale mixtures applied to multiscale image representations. PLoS One, 7(7), e39857, which seems particularly relevant to this paper.Typos: Line 79: “fixed” is repeated twice.Line 264: y_i should be y_1