  This paper investigates a phenomenon where query-based object detectors
mispredict at the last decoding stage while predicting correctly at an
intermediate stage. We review the training process and attribute the overlooked
phenomenon to two limitations: lack of training emphasis and cascading errors
from decoding sequence. We design and present Selective Query Recollection
(SQR), a simple and effective training strategy for query-based object
detectors. It cumulatively collects intermediate queries as decoding stages go
deeper and selectively forwards the queries to the downstream stages aside from
the sequential structure. Such-wise, SQR places training emphasis on later
stages and allows later stages to work with intermediate queries from earlier
stages directly. SQR can be easily plugged into various query-based object
detectors and significantly enhances their performance while leaving the
inference pipeline unchanged. As a result, we apply SQR on Adamixer, DAB-DETR,
and Deformable-DETR across various settings (backbone, number of queries,
schedule) and consistently brings 1.4-2.8 AP improvement.

Computer Vision and Pattern Recognition