  Many open-world applications require the detection of novel objects, yet
state-of-the-art object detection and instance segmentation networks do not
excel at this task. The key issue lies in their assumption that regions without
any annotations should be suppressed as negatives, which teaches the model to
treat the unannotated objects as background. To address this issue, we propose
a simple yet surprisingly powerful data augmentation and training scheme we
call Learning to Detect Every Thing (LDET). To avoid suppressing hidden
objects, background objects that are visible but unlabeled, we paste annotated
objects on a background image sampled from a small region of the original
image. Since training solely on such synthetically-augmented images suffers
from domain shift, we decouple the training into two parts: 1) training the
region classification and regression head on augmented images, and 2)~training
the mask heads on original images. In this way, a model does not learn to
classify hidden objects as background while generalizing well to real images.
LDET leads to significant improvements on many datasets in the open-world
instance segmentation task, outperforming baselines on cross-category
generalization on COCO, as well as cross-dataset evaluation on UVO and
Cityscapes.

Computer Vision and Pattern Recognition