Tree-Structured Reinforcement Learning for Sequential Object Localization
Existing object proposal algorithms usually search for possible object regions over multiple locations and scales separately, which ignore the interdependency among different objects and deviate from the human perception procedure. To incorporate global interdependency between objects into object localization, we propose an effective Tree-structured Reinforcement Learning (Tree-RL) approach to sequentially search for objects by fully exploiting both the current observation and historical search paths. The Tree-RL approach learns multiple searching policies through maximizing the long-term reward that reflects localization accuracies over all the objects. Starting with taking the entire image as a proposal, the Tree-RL approach allows the agent to sequentially discover multiple objects via a tree-structured traversing scheme. Allowing multiple near-optimal policies, Tree-RL offers more diversity in search paths and is able to find multiple objects with a single feedforward pass. Therefore, Tree-RL can better cover different objects with various scales which is quite appealing in the context of object proposal. Experiments on PASCAL VOC 2007 and 2012 validate the effectiveness of the Tree-RL, which can achieve comparable recalls with current object proposal algorithms via much fewer candidate windows.
This paper proposes a method for generating object proposals by using a sequential search strategy that is trained with reinforcement learning (Q-learning specifically). The paper has two major contributions: one, at test time, instead of taking just one action at each step, it takes two actions (one best action each from two classes of actions) and bifurcates. This allows the model to actually predict multiple objects and deal with the associated multimodality better. Two, the reward signal is carefully designed and while not imposing any order, provides rewards both when the localization of an object improves and when the model latches on to a new object. The recall achieved by the method seems impressive, and when plugged into the fast r-cnn improves over RPN by about 2 points. 
I liked the ideas present in the paper. This is the first sequential search strategy I have seen that tries to output all objects in an image, and does not impose any arbitrary and hard-to-justify order on the boxes during training. I have a minor clarification, and then some ways of strengthening the experimental section, which to me would be the difference between a poster and an oral.  A point of clarification: It seems that the ranking of the proposals output is simply the depth of the tree at which they are discovered. Why is this a good ranking? Is there anything in the training that encourages the model to discover objects quickly? I can think of some intuitive explanation but it would be good for the authors to provide an explanation in the paper.  Proposals are a fast moving subfield of computer vision and this paper would do well to add a few more experiments: 1) Comparison to more proposal methods. In particular, I would want comparisons to  MCG[1] and DeepMask[2], both of which are significantly better than, eg. edge boxes. MCG proposals are also available online 2) Comparison on COCO. COCO captures a much larger range of scale variations especially when it comes to small objects and methods like RPN typically do not fare very well. I would like to see how this fares. In addition, if the authors envisage this as a proposal method, then it should generalize to object categories that the network has not seen during training. It would be good to see if this is indeed true and if so, to what extent; COCO has a few categories that are not present either in PASCAL or Imagenet, so I would be curious how this method fares when trained on PASCAL and tested on COCO  It would also be nice to see what the detection AP looks like for higher overlap thresholds: this method seems to improve recall for higher IU so it might provide larger gains under more stringent evaluation metrics.