  Object detectors are conventionally trained by a weighted sum of
classification and localization losses. Recent studies (e.g., predicting IoU
with an auxiliary head, Generalized Focal Loss, Rank & Sort Loss) have shown
that forcing these two loss terms to interact with each other in
non-conventional ways creates a useful inductive bias and improves performance.
Inspired by these works, we focus on the correlation between classification and
localization and make two main contributions: (i) We provide an analysis about
the effects of correlation between classification and localization tasks in
object detectors. We identify why correlation affects the performance of
various NMS-based and NMS-free detectors, and we devise measures to evaluate
the effect of correlation and use them to analyze common detectors. (ii)
Motivated by our observations, e.g., that NMS-free detectors can also benefit
from correlation, we propose Correlation Loss, a novel plug-in loss function
that improves the performance of various object detectors by directly
optimizing correlation coefficients: E.g., Correlation Loss on Sparse R-CNN, an
NMS-free method, yields 1.6 AP gain on COCO and 1.8 AP gain on Cityscapes
dataset. Our best model on Sparse R-CNN reaches 51.0 AP without test-time
augmentation on COCO test-dev, reaching state-of-the-art. Code is available at
https://github.com/fehmikahraman/CorrLoss

Computer Vision and Pattern Recognition