Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
We present a framework for efficient inference in structured image models that explicitly reason about objects. We achieve this by performing probabilistic inference using a recurrent neural network that attends to scene elements and processes them one at a time. Crucially, the model itself learns to choose the appropriate number of inference steps. We use this scheme to learn to perform inference in partially specified 2D models (variable-sized variational auto-encoders) and fully specified 3D models (probabilistic renderers). We show that such models learn to identify multiple objects -counting, locating and classifying the elements of a scenewithout any supervision, e.g., decomposing 3D images with various numbers of objects in a single forward pass of a neural network at unprecedented speed. We further show that the networks produce accurate inferences when compared to supervised counterparts, and that their structure leads to improved generalization.
This paper presents a framework for "vision as inverse graphics" that combines structured generative models, inference networks, and end-to-end learning. A key element of the framework is the use of a trans-dimensional generative model that allows for the generation of a variable number of objects in the scene. The inference network that enables learning in the model uses a recurrent network to parse the scene one object at a time. This mechanism simultaneously parses the current object in terms of what/where information and predicts whether there is another object in the scene that remains to be parsed. The evaluation considers a number of 2D and 3D instances of the proposed model including a 2D model that directly learns object appearances and a 3D model that integrates a full 3D renderer. The representations learned by the framework are used for a variety of down-stream tasks including counting objects, where the performance is shown to outperform DRAW networks.  
Review Update: Tank you for the additional information on computation time as well as clarifying the issue with repetitions.  Technical quality: The technical contribution of the paper is defined at the level of a framework with modular parts and so is quite high-level as a result. The main components are the generative model (Eq 1), the recurrent inference network, and the use of variational learning. To the extent that technical details are provided for these components, they are correct.   The bulk of the paper focuses on the construction of experiments and the analysis of the results. In general, the data sets and tasks are well designed in both the 2D and 3D cases. The results in terms of object counts are promising. One aspect that could have been better evaluated in the 2D experiments is the accuracy of the pose (where) parameters inferred. This information is only presented qualitatively in terms of the reconstructions (although these do look good in the examples shown). In the case of 3D scenes, the results are also promising. However, the reconstruction accuracy metric appears not to be defined. The discussion of the problem caused by repetitions in the data sets is also not clear and should be explained further.   A final important point regarding the experiments is that the paper claims to enable scene understanding at unprecedented speeds, but there are no timing results of any kind in the paper. The video in the supplemental material appears to be showing real-time scene parsing for multiple digits, but the speed results need to be directly substantiated in the main paper.  Novelty/originality: As mentioned already, the paper describes a framework that combines several existing components: generative scene models, recurrent networks for inference, and variational learning. The type of generative model described has been studied in computer vision extensively in the vision as inverse graphics literature. This paper introduces a specific form of such a model that allows for inference of the number of objects and combines it with a recurrent inference network that can also reason about variable numbers of objects, which does appear to be a novel combination. The main novel claim made in the paper is enabling fast inference in this type of model, which has been the main issue in past work. However, as mentioned above, direct results substantiating the speed of the proposed framework are not included in the paper.  Impact: As a framework, this work is also very extensible in different directions. The use of graphical models combined with RNN inference networks to enable parsing scenes with multiple objects is a nice idea and is likely to be adopted by others if the speed claims are suitably substantiated.   Clarity and presentation: The paper is very well written. There are no issues with grammar or language. The paper is clear, although the descriptions are quite high-level for the most part. The figures are well done although most of the line graphs could benefit from being larger.