Hybrid Knowledge Routed Modules for Large-scale Object Detection
The dominant object detection approaches treat the recognition of each region separately and overlook crucial semantic correlations between objects in one scene. This paradigm leads to substantial performance drop when facing heavy long-tail problems, where very few samples are available for rare classes and plenty of confusing categories exists. We exploit diverse human commonsense knowledge for reasoning over large-scale object categories and reaching semantic coherency within one image. Particularly, we present Hybrid Knowledge Routed Modules (HKRM) that incorporates the reasoning routed by two kinds of knowledge forms: an explicit knowledge module for structured constraints that are summarized with linguistic knowledge (e.g. shared attributes, relationships) about concepts; and an implicit knowledge module that depicts some implicit constraints (e.g. common spatial layouts). By functioning over a region-to-region graph, both modules can be individualized and adapted to coordinate with visual patterns in each image, guided by specific knowledge forms. HKRM are light-weight, general-purpose and extensible by easily incorporating multiple knowledge to endow any detection networks the ability of global semantic reasoning. Experiments on large-scale object detection benchmarks show HKRM obtains around 34.5% improvement on VisualGenome (1000 categories) and 30.4% on ADE in terms of mAP. Codes and trained model can be found in https://github.com/chanyn/HKRM.
This paper is about an object detection method that exploits both implicit and explicit knowledge to increase the semantic consistency of detections in a scene. Starting from one of the current state of art works in object detection (Faster-RCNN) this approach extract object proposal from its RPN, and produce an augmented feature where explicit and implicit knowledge, in the form of feature graphs, is integrated before producing the final classification and bounding box regression.  While exploiting the knowledge of spatial, similarity, attributes and context has a long history in the literature (the work cite several of them), this work has the merit of producing adaptive region-to-region graphs and put its foundation on a modern object detection framework that is known to be more reliable than in the past. Moreover the use of Visual Genome and the scene parsing dataset ADE enable the possibility of seeing a marked improvement that was probably not possible in the past.  Strengths: + Shows a framework able to integrate explicit and implicit knowledge in a modern framework for object detection such as Faster-RCNN. + Very well written and easy to read. + Enables further studies among this important line of research.  Weakness: - The method is tested on ADE and Visual Genome while all previous work on object detection typically use COCO, Pascal VOC 2007 or ILSRVC detection challenge datasets. While they may be considered less challenging than ADE and Visual Genome, it would have been interesting to observe if there is any advantage in using the encoded knowledge in such more simplified dataset. Moreover, it would enable comparison with other state of the art works. - As a continuation of the previous point, ADE is a dataset for Image parsing, which may be biased among different scenes and favour the performance of the proposed method. - The approach is claimed to be "light-weight, general-purpose and extensible by easily incorporating multiple knowledge to endow any detection networks the ability of global semantic reasoning.". Beside Faster-RCNN, since it is an uncommon dataset for object detection, it would be interesting to see more comparisons with other state of the art methods that have better performance than Faster-RCNN. For instance, YOLOv3 and SSD. Such other methods may already have better performance than the proposed method without using any knowledge. Even in such a case, it would not decrease the significance of the paper, but would put more in focus the actual performance that is obtainable by exploiting the knowledge by the proposed method. - The novelty of integrating knowledge is not new.  Suggestion: - I would like to suggest the authors to release the source code and/or the trained models.  Very minor issues: - row 215: Faser-RCNN -> Faster-RCNN - Faster-rcnn is written differently among the paper.  —————————- Thanks to the authors for addressing my concerns. They are all well addressed in the rebuttal.