Current Convolutional Neural Network (CNN)-based object detection models adopt strictly feedforward inference to predict the final detection results. However, the widely used one-way inference is agnostic to the global image context and the interplay between input image and task semantics. In this work, we present a general technique to improve off-the-shelf CNN-based object detection models in the inference stage without re-training, architecture modification or ground-truth requirements. We propose an iterative, bottom-up and top-down inference mechanism, which is named conscious inference, as it is inspired by prevalent models for human consciousness with top-down guidance and temporal persistence. While the downstream pass accumulates category-specific evidence over time, it subsequently affects the proposal calculation and the final detection. Feature activations are updated in line with no additional memory cost. Our approach advances the state of the art using popular detection models (Faster-RCNN, YOLOv2, YOLOv3) on 2D object detection and 6D object pose estimation.
The paper proposes a method called cautious inference to improve inference accuracy for object detection models. The main idea is inspired by the previous work of Guided Perturbations, which is applied to fully convolutional networks to improve the segmentation/accuracy accuracy purely during inference time.  The original idea is to use the predicted labels of the network as pseudo ground truths (after making the predictions to be a one-hot vector), and then back propagate the error signals to the network input to get the gradients. And finally the gradients are added back to the original inputs to perform another round of prediction. Here the inputs can be either the original image, or some intermediate feature maps. Experiments are shown for both 2D and 6D object detections. 
Comments:
- I think overall it is an interesting idea to directly alter the input of the network in order to fit to the testing distribution. However, the motivation and story told in the introduction is a bit of an oversell compared to the experiment validation section. Most of the results shown are just doing training and testing of images drawn from the *same* distribution. Like coco train and test, or VOC train and test. It would be great to see if the cautious inference would work when the distribution is different. For example "elephant in the room" case, or new object categories are added during testing.
- I am actually curious to see this method can be used to improve the AP on the *training* set as well, just to understand it better -- is it trying to recover the generalization error of the network, or it is doing some implicit context reasoning inference that can help training as well. 
- It might be better to compare/combine the method to other inference-only improvements for object detection. For example there is soft-NMS, 
Bodla, Navaneeth, et al. "Soft-nms—improving object detection with one line of code." Computer Vision (ICCV), 2017 IEEE International Conference on. IEEE, 2017.
 - I am not sure I fully understand B-box part: I think it is easy to have B-obj and B-cls as one can just take the max of the class prediction and then use the inferred class label for one-hot vector construction, but I am confused about the box part as no ground-truth is given during testing. In Table 2 I also cannot find BP improving performance by itself in anyway.
- For COCO, please report results on test-dev set, the minival set images are used only for validation.