FEATURE INTERTWINER FOR OBJECT DETECTION
A well-trained model should classify objects with a unanimous score for every category. This requires the high-level semantic features should be alike among samples, despite a wide span in resolution, texture, deformation, etc. Previous works focus on re-designing the loss function or proposing new regularization constraints on the loss. In this paper, we address this problem via a new perspective. For each category, it is assumed that there are two sets in the feature space: one with more reliable information and the other with a less reliable source. We argue that the reliable set could guide the feature learning of the less reliable set during training -in the spirit of student mimicking teacher's behavior and thus pushing towards a more compact class centroid in the high-dimensional space. Such a scheme also benefits the reliable set since samples become closer within the same category -implying that it is easier for the classifier to identify. We refer to this mutual learning process as feature intertwiner and embed the spirit into object detection. It is well-known that objects of low resolution are more difficult to detect due to the loss of detailed information during network forward pass. We thus regard objects of high resolution as the reliable set and objects of low resolution as the less reliable set. Specifically, an intertwiner is achieved by minimizing the distribution divergence between two sets. We design a historical buffer to represent all previous samples in the reliable set and utilize them to guide the feature learning of the less reliable set. The design of obtaining an effective feature representation for the reliable set is further investigated, where we introduce the optimal transport (OT) algorithm into the framework. Samples in the less reliable set are better aligned with the reliable set with aid of OT metric. Incorporated with such a plug-and-play intertwiner, we achieve an evident improvement over previous state-of-the-arts on the COCO object detection benchmark.
OVERVIEW:
The authors tackle the problem of detecting small/low resolution objects in an image. Their key idea is that detecting bigger objects is an easier task and can be used to guide the detection of smaller objects. This is done using the "Feature Intertwiner"  which consists of two branches, one for the larger objects (more reliable set that is also easier to detect) and one for the smaller objects (less reliable set). The second branch contains a make-up layer learned during training (which acts as the guidance from the more reliable set) that helps compensate details needed for detection. The authors define a class buffer that contains representative elements of object features from the reliable set for every category & scale and an intertwiner loss that computes the L2 loss between the features from the less reliable set & the class buffer. They also use an Optimal Transport procedure with a Sinkhorn divergence loss between object features from both sets. The overall loss of the system is now a sum of the detection loss, the intertwiner loss and the optimal transport loss. They evaluate their model on the COCO Object detection challenge showing state-of-the-art performance. They also provide thorough ablation analysis of various design choices. The qualitative result in Fig.1 showing well clustered features for both high & low resolution objects via t-SNE is a nice touch.
COMMENTS:
Clarity - The paper is well written and easy to follow.
Originality & Significance - The paper tackles an important problem and provides a novel solution. 
Quality - The paper is complete in that it tackles an important problem, provides a novel solution and demonstrates via thorough experiments the improvement achieved using their approach. 
QUESTIONS:
1. The Class Buffer seems very restricted in having a single element per object category per scale to represent all features. The advantage of forcing such a representation is tight clustering in the feature space. But, wouldn't a dictionary approach with multiple elements give more flexibility to the model and learn a richer feature representation at the cost of not-so-good clustering ?
2. Any comment on why you drop performance for couch ? (and baseball bat + bedroll)
3. In Table 4 of Appendix where you compare with more object detection results, I find it interesting that Mask RCNN, updated results has a might higher AP_S (43.5) compared to you (27.2) and everyone else. I was expecting you to be the best under that metric due to the explicit design for small objects. They (MaskRCNN, updated results) are also significantly better than the rest under AP_M but worse under AP_L. Can you explain this behavior ? Is the ResNeXt backbone that much better for small objects ?