Generative Image Modeling Using Spatial LSTMs
Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels. Recurrent neural networks have been successful in capturing long-range dependencies in a number of problems but only recently have found their way into generative image models. We here introduce a recurrent image model based on multidimensional long short-term memory units which are particularly suited for image modeling due to their spatial structure. Our model scales to images of arbitrary size and its likelihood is computationally tractable. We find that it outperforms the state of the art in quantitative comparisons on several image datasets and produces promising results when used for texture synthesis and inpainting.
The basic idea of this paper is to replace the MCGSM (mixture of conditional Gaussian scale mixtures) of [38] with a version where a continuous-valued hidden state vector h is maintained in a LSTM. This is used as a model of natural images and assessed by a density estimation task (secs 3.2 and 3.3, Tables 1-3), and for texture synthesis and inpainting (sec 3.4).The model for p(x_ij|h_ij) (l 150) is in fact not specified at all. Given that h is a continuous-valued vector (es per eq 6) we need to see some functional form. RNADE [41] is designed for fixed-length vectors. In the case of "infinite dimensional images" it would be very simple to build a mixture density network (MDN) or mixture of experts (see e.g. Bishop's 1995 book sec 6.4) to predict x_{ij} given a finite window of context (as per Fig 1A); (R)NADE is used in the more complex case of fixed-length data vectors. This comparison needs to be made to properly evaluate the benefit of RIDE in the Tables, as the MCGSM (as per eqs 4 and 5) is a relatively simple MDN. RIDE has theoretical benefits of potentially using infinite context as opposed to finite context, cf FIR vs IIR filters, but this theoretical benefit needs to be demonstrated in practice.RIDE is compared with variants of the MCGSM on "infinite dimensional images", and as per Tables 1 and 2 (RH column) it shows small advantages over MCGSM. However, as per the above para, we need to see comparisons with a MDN which is richer than a MCGSM (e.g. using hidden layers).The authors claim (l 371) that inpainting has only been done for small image regions until recently (citing [32] from 2015).However, there is work before this inpainting large regions, see e.g.Multiple Texture Boltzmann Machines, J. J. Kivinen and Christopher K. I. Williams, Proceedings AISTATS 2012, http://homepages.inf.ed.ac.uk/ckiw/postscript/mtbm.pdf . In that paper the authors consider the obvious evaluation of inpainting, computing the normalized cross correlation (NCC) of the inpainted region with the ground truth; this method should be used toevaluate the RIDE model quantitatively. Also in the same paper the authors used a texture similarity score (TSS) for assessing thequality of texture samples. Quality: The paper makes sense technically, using the spatial LSTM model of [8] to model images.Clarity: Writing is clear, except for defn of p(x_ij|h_ij) (l 150).Originality: A relatively straightforward application of the spatial LSTM framework (although getting it all to work well would require a lot of effort unless a good spatial LSTM code base is to hand).Significance: As per the comments above, we need to see (1) comparisonwith a MDN predictor (with careful effort to make it work as well as it can, as you have done for LSTM), and (2) better comparisons for texture synthesis/inpainting.Overall: Marginally below threshold due to lack of comparisons for density estimation and texture work.Other points:l 135 provide name for g units.l 280 Table 3 Aveage -> Averagel 312. The "dead leaves" model was introduced by Matheron in 1968, and further developed by D. Jeulin, see e.g.Jeulin, D. (1989). Morphological modeling of images by sequential random functions. Signal Processing 16, 403-431.Jeulin, D. (1996). Dead leaves models: From space tesselation to random functions. In Advances in Theory and Applications of Random Sets. ed. D. Jeulin. pp. 137-156.l 359 -- give url for where you obtained Brodatz data.ADDED AFTER REBUTTAL: =====================The other reviews and the author response have been read. The author clarification that the model for p(x_ij|h_ij) is a MCGSM means that the results are making a fair comparison between a MCGSM for p(x_ij|x_{< ij}) and p(x_ij|h_ij).I have thus raised my score to 6.I still believe that using a more complex model than a MCGSM (with hidden layers) may lead to benefits and encourage the authors topursue this line of enquiry.