Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
Textual grounding is an important but challenging task for human-computer interaction, robotics and knowledge mining. Existing algorithms generally formulate the task as selection from a set of bounding box proposals obtained from deep net based systems. In this work, we demonstrate that we can cast the problem of textual grounding into a unified framework that permits efficient search over all possible bounding boxes. Hence, the method is able to consider significantly more proposals and doesn't rely on a successful first stage hypothesizing bounding box proposals. Beyond, we demonstrate that the trained parameters of our model can be used as word-embeddings which capture spatial-image relationships and provide interpretability. Lastly, at the time of submission, our approach outperformed the current state-of-the-art methods on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77% respectively.
Summary
The approach proposes a simple method to find the globally optimal (under the formulation) box in an image which represents the grounding of the textual concept. Unlike previous approaches which adopt a two stage pipeline where region proposals are first extracted and then combined into grounded image regions (see [A] for example), this approach proposes a formulation which finds the globally optimal box for a concept. The approach assumes the presence of spatial heat maps depicting various concepts, and uses priors and geometry related information to construct an energy function, with learnable parameters for combining different cues. A restriction of the approach is that known concepts can only be combined linearly with each other (for instance the score map of “dirt bike” is “dirt” + “bike” with the learn weighting ofcourse), but this also allows for optimal inference for the given model class. More concretely the paper proposes an efficient technique based on [22] to use branch and bound for efficient sub-window search. Training is straightforward and clean through cutting-plane training of structured SVM. The paper also shows how to do efficient loss augmented inference during SVM training which makes the same branch and bound approach applicable to cutting-plane training as well. Finally, results are shown against competitive (near- state of the art approaches) on two datasets where the proposed approach is shown to outperform the state of the art.
Strengths
- Approach alleviates the need for a blackbox stage which generates region proposals.
- The interpretation of the weights of the model and the concepts as word embeddings is a neat little tidbit.
- The paper does a good job of commenting on cases where the approach fails, specifically pointing out some interesting examples such as “dirt bike” where the additive nature of the feature maps is a limitation.
- The paper has some very interesting ideas such as the use of the classic integral images technique to do efficient inference using branch and bound, principled training of the model via. a clean application of Structural SVM training with the cutting plane algorithm etc.
Weakness
1. Paper misses citing a few relevant recent related works [A], [B], which could also benefit from the proposed technique and use region proposals.
2. Another highly relevant work is [C] which does efficient search for object proposals in a similar manner to this approach building on top of the work of Lampert et.al.[22]
3. It is unclear what SPAT means in Table. 2.
4. How was Fig. 6 b) created? Was it by random sub-sampling of concepts?
5. It would be interesting to consider a baseline which just uses the feature maps (used in the work, say shown  in Fig. 2) and the phrases and simply regresses to the target coordinates using an MLP. Is it clear that the proposed approach would outperform it? (*)
6. L130: It was unclear to me how the geometry constraints are exactly implemented in the algorithm, i.e. the exposition of how the term k2 is computed was uncler. It would be great to provide details. Clear explanation of this seems especially important since the performance of the system seems highly dependent on this term (as it is trivial to maximize the sum of scores of say detection heat maps by considering the entire image as the set).
Preliminary Evaluation
The paper has a neat idea which is implemented in a very clean manner, and is easy to read. Concerns important for the rebuttal are marked with (*) above.
[A] Hu, Ronghang, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. 2016. “Modeling Relationships in Referential Expressions with Compositional Modular Networks.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1611.09978.
[B] Nagaraja, Varun K., Vlad I. Morariu, and Larry S. Davis. 2016. “Modeling Context Between Objects for Referring Expression Understanding.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1608.00525.
[C] Sun, Qing, and Dhruv Batra. 2015. “SubmodBoxes: Near-Optimal Search for a Set of Diverse Object Proposals.” In Advances in Neural Information Processing Systems 28, edited by C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, 1378–86. Curran Associates, Inc.