Multi-view recognition is the task of classifying an object from multi-view image sequences. Instead of using a single-view for classification, humans generally navigate around a target object to learn its multi-view representation. Motivated by this human behavior, the next best view can be learned by combining object recognition with navigation in complex environments. Since deep reinforcement learning has proven successful in navigation tasks, we propose a novel multi-task reinforcement learning framework for joint multi-view recognition and navigation. Our method uses a hierarchical action space for multi-task reinforcement learning. The framework was evaluated with an environment created from the ModelNet40 dataset. Our results show improvements on object recognition and demonstrate human-like behavior on navigation.
The ambition of this paper is to address multi-view object recognition and the associated navigation as a unified reinforcement learning problem using a deep CNN to represent the policy.
Multi-view recognition and active viewpoint selection have been studied for more than 30 years, but this paper ignores most of this history.  The discussion of related work as well as the empirical evaluation are limited to very recent methods using neural networks.  I encourage the authors to look e.g. at Paletta and Pinz [1] (who solve a very similar and arguably harder problem in related ways) and at Bowyer & Dyer [2] as well as the references contained in these papers for history and context.  Active vision goes back to Bajcsy, Aloimonos, and Ballard; these should be cited instead of Ammirato et al.  Conversely, the related work cites a handful of papers (e.g. in the context of Atari 2600 games) that are unrelated to this work.
The navigation aspect is limited to fixed-size left or right displacements (at least for ModelNet40 task which is the only one to be evaluated and discussed).  This is strictly weaker than active viewpoint selection.  Adding this to the disregard of prior work, it is (at best) misleading to claim that this is "the first framework to combine learning of navigation and object recognition".
Calling this "multi-task" learning is also misleading.  There is only one ultimate objective (object recognition), while the agent has two types of actions available (moving or terminating with a classification decision).
There are other misleading, vague, or inaccurate statements in the paper, for example:
- "With the introduction of deep learning to reinforcement learning, there has been ... advancements in understanding ... how humans navigate": I don't think such a link exists; if it does, a citation needs to be provided.
- "inductive bias like image pairs": Image pairs do not constitute inductive bias.  Either the term is misused or the wording must be clarified; likewise for other occurrences of "inductive bias".
- "a single softmax layer is biased towards tasks with larger number of actions": I think I understand what this is intended to say, but a "softmax layer" cannot be "biased towards tasks" as there is only one, given, task.
- I do not understand what the stated contribution of "extrapolation of the action space to a higher dimension for multi-task learning" is meant to be.
- "Our method performs better ... than state-of-the-art in training for navigation to the object": The method does not involve "navigation to the object", at least not for the ModelNet40 dataset, the only for which results are given.
It is not clear what objective function the system is intended to optimize.  Since the stated task is object recognition and from Table 2 I was expecting it to be the misclassification rate, but this is clearly not the case, as the system is not set up to minimize it.  What "biases" the system towards classification actions (p. 5)?  Why is it bad if the agent shows "minimal movement actions" as long as the misclassification rate is minimized? No results are given to show whether this is the case or not.  The text then claims that the "hierarchical method gives superior results", but this is not shown either.
Table 3 reveals that the system fails to learn much of interest at all.  Much of the time the agent chooses not to move and performs relatively poorly; taking more steps improves the results; often all 12 views are collected before a classification decision is made.  Two of the most important questions remain open: (1) What would be the misclassification rate if all views are always used? (2) What would be the misclassification rate under a random baseline policy not involving navigation learning (e.g., taking a random number of steps in the same direction)?
Experiments using the THOR dataset are announced but are left underspecified (e.g., the movement actions), but no results or discussion are given.
SUMMARY
Quality: lacking in may ways; see above.
Clarity: Most of the paper is clear enough, but there are confusions and missing information about THOR and problems with phrasing and terminology.  Moreover, there are many grammatical and typographical glitches.
Originality: Harder tasks have been looked at before (using methods other than CNN).  Solving a simpler version using CNN I do not consider original unless there is a compelling pay-off, which this paper does not provide.
Significance: Low.
Pros: The problem would be very interesting and relevant if it was formulated in a more ambitious way (e.g., a more elaborate action space than that used for ModelNet40) with a clear objective function,
Cons: See above.
[1] Lucas Paletta and Axel Pinz, Active object recognition by view integration and reinforcement learning, Robotics and Autonomous Systems 31, 71-86, 2000
[2] Bowyer, K. W. and Dyer, C. R. (1990), Aspect graphs: An introduction and survey of recent results. Int. J. Imaging Syst. Technol., 2: 315–328. doi:10.1002/ima.1850020407