We show how discrete objects can be learnt in an unsupervised fashion from pixels, and how to perform reinforcement learning using this object representation. More precisely, we construct a differentiable mapping from an image to a discrete tabular list of objects, where each object consists of a differentiable position, feature vector, and scalar presence value that allows the representation to be learnt using an attention mechanism. Applying this mapping to Atari games, together with an interaction net-style architecture for calculating quantities from objects, we construct agents that can play Atari games using objects learnt in an unsupervised fashion. During training, many natural objects emerge, such as the ball and paddles in Pong, and the submarine and fish in Seaquest. This gives the first reinforcement learning agent for Atari with an interpretable object representation, and opens the avenue for agents that can conduct objectbased exploration and generalization.
This paper learns to construct masks and feature representations from an input image, in order to represent objects. This is applied to the relatively simple domain of Atari games video input (compared to natural images). The paper is completely inadequate in respect to related work; it re-invents known techniques like non-maximum suppression and matching for tracking; fails to learn convincing objects according to visual inspection; and fails to compare with earlier methods for these tasks. (The comment above about re-invention is the most charitable intepretation -- the worst case would be using these ideas without citation.)
1) The related work section is outrageous, containing no references before 2016.  Do the authors think researchers never tried to do this task before then? This is the bad side of the recent deep nets hype, and ICLR is particularly susceptible to this. Examples include
@article{wang-adelson-94,
  author        = "Wang,  J. Y. A. and Adelson, E. H.",
  title         = {{Representing Moving Images with Layers}},
  journal       = {{IEEE Transactions on Image Processing}},
  year          = "1994",
  volume        = "3(5)",
  pages         = {625-638}
}
see http://persci.mit.edu/pub_pdfs/wang_tr279.pdf
and
@article{frey-jojic-03,
   author    = {Frey, B. J. and Jojic, N.},
   title     = {{Transformation Invariant Clustering Using the EM Algorithm}},
   journal   = {IEEE Trans Pattern Analysis and Machine Intelligence},
   year      = {2003},
   volume    = {25(1)},
   pages     = {1-17}
}
where mask and appearances for each object of interest are learned. There is a literature which follows on from the F&J paper.  The methods used in Frey & Jojic are different from what is proposed in the paper, but there needs to be comparisons.
The AIR paper also contains references to relevant previous work.
2) p 3 center -- this seems to be reinventing non-maximum suppression
3) p 4 eq 3 and sec 3.2 -- please justify *why* it makes sense to use
the concrete transform.  Can you explain better (e.g. in the supp mat)
the effect of this for different values of q_i?
4) Sec 3.5 Matching objects in successive frames using the Hungarian 
algorithm is also well known, e.g. it is in the matlab function
assignDetectionsToTracks .
5) Overall: in this paper the authors come up with a method for learning objects from Atari games video input. This is a greatly restricted setting compared to real images. The objects learned as shown in Appendix A are quite unconvincing, e.g. on p 9. For example for Boxing why are the black and white objects broken up into 3 pieces, and why do they appear coloured in col 4?
Also the paper lacks comparisons to other methods (including ones from before 2016) which have tackled this problem.
It may be that the methods in this paper can outperform previous ones -- that would be interesting, but it would need a lot of work to address the issues raised above.
Text corrections:
p 2 "we are more precise" -> "we give more details"
p 3 and p 2 -- local maximum (not maxima) for a single maximum.  [occurs many times]