summaries: The paper addresses the problem of open world object detection - a lifelong learning system where the object detector is required to detect objects belonging to all classes known so far and “unknown” objects. Unknown objects are annotated and available for training at a later stage. At a given time only a specific set of training data is available with annotations that involve all known classes so far, while all past training data are unavailable.    The proposed approach builds upon recent work [A], by incorporating a semantic topology in the object feature space (RoI features in Faster R-CNN). While [A] use contrastive clustering to group object features based on object category, and energy based out-of-distribution detection to detect “unknown” objects, the authors propose an end-to-end trainable approach. The proposed approach uses RoI features in two parallel streams, with one focused on clustering operation, while having object category prediction loss on both streams of features. Features are clustered around predetermined semantically meaningful object representations. Pre-trained word embeddings from a language model (CLIP) are used for this purpose. The use of pre-defined semantic anchors allows learned features of known objects to form consistent clusters during incremental learning.    Experiments are conducted using 80 categories from PASCAL-VOC and MSCOCO datasets. The problem is designed as 4 sequential tasks each involving 20 object categories (same setup as in [A]). Detection performance is measured using mAP, as well as measures that focus on unknown object detection.  [A] Towards Open World Object Detection, Joseph et al., CVPR 2021   This paper extends the recently-introduced object ORE object detector [A]. ORE is trained in an incremental fashion and was shown to minimize the confusion between classes presented in different task sets by adding a contrastive objective to the model training, that pushes features, representing different semantic classes, far away in the feature space. This paper presents the case that combining ORE detector with large-scale language models, trained by aligning textual queries and images, can significantly improve ORE detector results. This is not surprising: it was already shown that such multi-modal language models could be successfully used for detection of novel classes in a zero-shot setting (in which names of target classes are given, but image data for these classes is available during the model training, see [D]). To the best of my knowledge, the core idea of semantic anchoring using language models was introduced in [C] (see Sec. 3.3 Densely Sampled Embedding Space). In this work, the semantic embedding space is extended with additional data from external sources that contain semantic information about the unseen classes via language embedding to image embedding alignment. This method suggests using CLIP [B] instead (which makes sense and simplifies the knowledge transfer!)  [A] Joseph et al., CVPR’21 [B] Radford et al., ICLR’21 [C] Bansal et al., ECCV’18 [D] Xian et al., CVPR'17 [E] Zheng et al., CVPR’21 This paper proposes a semantic topology embedding for Open-World Object Detection (OWOD) where an object detector identifies objects of unknown classes and incrementally learns to classify them assuming that their annotations are progressively given by humans. To maintain discriminative and consistent relationships among object classes, the authors introduce a semantic topology for the feature space of the detector by constructing pre-deﬁned anchors for categories using a pretrained language model. During training, it enables the detector to distinguish unknown objects out of the known categories and also makes learned features of different classes undistorted during incremental learning. Eperimental results show that the semantic topology improves state-of-the-art open-world object detectors and help the open-world detectors preserve a discriminative and consistent feature space. The paper proposes a novel method for open-world object detection, where instances of unknown categories need to be identified and annotated data for such new categories need to be integrated into the model in an incremental fashion. Prior work typically separate this problem into two tasks, out-of-distribution detection and incremental/continuous learning. This paper proposes to use fixed semantic anchors for each category, which are embedding vectors from a language model (or randomly generated vectors) for each category.  Importantly, when new data arrives, new embeddings are added while previous ones do not change. This encourages the feature representation to be compact (discriminative) and consistent (over time).
abstractText: A more realistic object detection paradigm, Open-World Object Detection, has arised increasing research interests in the community recently. A qualified openworld object detector can not only identify objects of known categories, but also discover unknown objects, and incrementally learn to categorize them when their annotations progressively arrive. Previous works rely on independent modules to recognize unknown categories and perform incremental learning, respectively. In this paper, we provide a unified perspective: Semantic Topology. During the life-long learning of an open-world object detector, all object instances from the same category are assigned to their corresponding pre-defined node in the semantic topology, including the ‘unknown’ category. This constraint builds up discriminative feature representations and consistent relationships among objects, thus enabling the detector to distinguish unknown objects out of the known categories, as well as making learned features of known objects undistorted when learning new categories incrementally. Extensive experiments demonstrate that semantic topology, either randomly-generated or derived from a well-trained language model, could outperform the current state-of-the-art open-world object detectors by a large margin, e.g., the absolute open-set error (the number of unknown instances that are wrongly labeled as known) is reduced from 7832 to 2546, exhibiting the inherent superiority of semantic topology on open-world object detection.
authors.name: SEMANTIC TOPOLOGY
authors.name: Shuo Yang
authors.name: Peize Sun
authors.name: Yi Jiang
authors.name: Xiaobo Xia
authors.name: Ruiheng Zhang
authors.name: Zehuan Yuan
authors.name: Changhu Wang
authors.name: Ping Luo
authors.name: Min Xu
references.authors: Ankan Bansal Karan Sikka Gaurav Sharma Rama Chellappa Ajay Divakaran
references.title: Zero-shot object detection
references.authors: Abhijit Bendale Terrance E Boult
references.title: Towards open set deep networks
references.venue: In CVPR,
references.authors: Zhaowei Cai Nuno Vasconcelos
references.title: Cascade R-CNN: Delving into high quality object detection
references.authors: Nicolas Carion Francisco Massa Gabriel Synnaeve Nicolas Usunier Alexander Kirillov Sergey Zagoruyko
references.title: End-to-End object detection with transformers
references.venue: In ECCV,
references.authors: Arslan Chaudhry Marc’Aurelio Ranzato Marcus Rohrbach Mohamed Elhoseiny
references.title: Efficient lifelong learning with a-gem
references.venue: arXiv preprint arXiv:1812.00420,
references.authors: Karan Desai Justin Johnson
references.title: Virtex: Learning visual representations from textual annotations
references.venue: In CVPR,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: pre-training of deep bidirectional transformers for language understanding
references.venue: In NAACL-HLT,
references.authors: Akshay Dhamija Manuel Gunther Jonathan Ventura Terrance Boult
references.title: The overlooked elephant of object detection: Open set
references.venue: In WACV,
references.authors: Akshay Raj Dhamija Manuel Günther Jonathan Ventura Terrance E. Boult
references.title: The overlooked elephant of object detection: Open set
references.venue: In WACV,
references.authors: Kaiwen Duan Song Bai Lingxi Xie Honggang Qi Qingming Huang Qi Tian
references.title: CenterNet: Keypoint triplets for object detection
references.authors: Mark Everingham Luc. Van Gool Christopher K.I. Williams John Winn Andrew Zisserman
references.title: The pascal visual object classes (VOC) challenge
references.authors: Pedro Felzenszwalb Ross Girshick David McAllester Deva Ramanan
references.title: Object detection with discriminatively trained part based models
references.authors: Andrea Frome Greg Corrado Jonathon Shlens Samy Bengio Jeffrey Dean Marc’Aurelio Ranzato Tomas Mikolov
references.title: Devise: A deep visual-semantic embedding model
references.authors: Ross Girshick
references.title: Fast R-CNN
references.venue: In ICCV,
references.authors: Ross Girshick Jeff Donahue Trevor Darrell Jitendra Malik
references.title: Rich feature hierarchies for accurate object detection and semantic segmentation
references.venue: In CVPR,
references.authors: Ben Goertzel
references.title: Artificial general intelligence: Concept, state of the art, and future prospects
references.venue: J. Artif. Gen. Intell.,
references.authors: Xiuye Gu Tsung-Yi Lin Weicheng Kuo Yin Cui
references.title: Zero-shot detection via vision and language knowledge distillation
references.venue: CoRR, abs/2104.13921,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.authors: Borui Jiang Ruixuan Luo Jiayuan Mao Tete Xiao Yuning Jiang
references.title: Acquisition of localization confidence for accurate object detection
references.authors: K J Joseph Salman Khan Fahad Shahbaz Khan Vineeth N Balasubramanian
references.title: Towards open world object detection
references.venue: In CVPR,
references.authors: KJ Joseph Vineeth N Balasubramanian
references.title: Meta-consolidation for continual learning
references.venue: arXiv preprint arXiv:2010.00352,
references.authors: Armand Joulin Laurens Van Der Maaten Allan Jabri Nicolas Vasilache
references.title: Learning visual features from large weakly supervised data
references.venue: In European Conference on Computer Vision,
references.authors: Ang Li Allan Jabri Armand Joulin Laurens van der Maaten
references.title: Learning visual n-grams from web data
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Zhizhong Li Derek Hoiem
references.title: Learning without forgetting, 2017
references.authors: Tsung-Yi Lin Michael Maire Serge Belongie James Hays Pietro Perona Deva Ramanan Piotr Dollár C. Lawrence Zitnick
references.title: Microsoft COCO: Common objects in context
references.venue: In ECCV,
references.authors: Tsung-Yi Lin Priya Goyal Ross Girshick Kaiming He Piotr Dollar
references.title: Focal loss for dense object detection
references.venue: In ICCV,
references.authors: Wei Liu Dragomir Anguelov Dumitru Erhan Christian Szegedy Scott Reed Cheng-Yang Fu Alexander C. Berg
references.title: SSD: Single shot multibox detector
references.venue: In ECCV,
references.authors: Weitang Liu Xiaoyun Wang John Owens Yixuan Li
references.title: Energy-based out-of-distribution detection
references.venue: NeurIPS,
references.authors: Weiyang Liu Yandong Wen Zhiding Yu Meng Yang
references.title: Large-margin softmax loss for convolutional neural networks
references.venue: In ICML,
references.authors: Ziwei Liu Zhongqi Miao Xiaohang Zhan Jiayun Wang Boqing Gong Stella X Yu
references.title: Large-scale long-tailed recognition in an open world
references.venue: In CVPR,
references.authors: Mario Livio
references.title: Why?: What makes us curious
references.authors: David Lopez-Paz Marc’Aurelio Ranzato
references.title: Gradient episodic memory for continual learning
references.venue: NeurIPS,
references.authors: John A Meacham
references.title: Wisdom and the context of knowledge: Knowing that one doesn’t know
references.venue: On the development of developmental psychology,
references.authors: Dimity Miller Lachlan Nicholson Feras Dayoub Niko Sünderhauf
references.title: Dropout sampling for robust object detection in open-set conditions
references.venue: In ICRA,
references.authors: Dimity Miller Lachlan Nicholson Feras Dayoub Niko Sünderhauf
references.title: Dropout sampling for robust object detection in open-set conditions
references.venue: In ICRA,
references.authors: Dimity Miller Lachlan Nicholson Feras Dayoub Niko Sünderhauf
references.title: Dropout sampling for robust object detection in open-set conditions
references.venue: In ICRA,
references.authors: Dimity Miller Niko Sünderhauf Michael Milford Feras Dayoub
references.title: Uncertainty for identifying open-set errors in visual object detection, 2021
references.authors: Can Peng Kun Zhao Brian C. Lovell
references.title: Faster ILOD: incremental learning for object detectors based on faster RCNN
references.authors: Alec Radford Jong Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever
references.title: Learning transferable visual models from natural language supervision. 2021a
references.authors: Alec Radford Jong Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark
references.title: Learning transferable visual models from natural language supervision
references.venue: arXiv preprint arXiv:2103.00020,
references.authors: Jathushan Rajasegaran Salman Khan Munawar Hayat Fahad Shahbaz Khan Mubarak Shah
references.title: itaml: An incremental task-agnostic meta-learning approach
references.venue: In CVPR,
references.authors: Sylvestre-Alvise Rebuffi Alexander Kolesnikov Georg Sperl Christoph H. Lampert
references.title: icarl: Incremental classifier and representation learning, 2017
references.authors: Joseph Redmon Ali Farhadi
references.title: YOLOv3: An incremental improvement
references.venue: arXiv preprint arXiv:1804.02767,
references.authors: Joseph Redmon Santosh Divvala Ross Girshick Ali Farhadi
references.title: You only look once: Unified, real-time object detection
references.authors: Shaoqing Ren Kaiming He Ross Girshick Jian Sun
references.title: Faster R-CNN: Towards real-time object detection with region proposal networks
references.venue: NeurIPS,
references.authors: Konstantin Shmelkov Cordelia Schmid Karteek Alahari
references.title: Incremental learning of object detectors without catastrophic forgetting
references.venue: In ICCV,
references.authors: Konstantin Shmelkov Cordelia Schmid Karteek Alahari
references.title: Incremental learning of object detectors without catastrophic forgetting
references.venue: In ICCV,
references.authors: Peize Sun Yi Jiang Enze Xie Zehuan Yuan Changhu Wang Ping Luo
references.title: Onenet: Towards end-to-end one-stage object detection
references.venue: arXiv preprint arXiv:2012.05780,
references.authors: Peize Sun Rufeng Zhang Yi Jiang Tao Kong Chenfeng Xu Wei Zhan Masayoshi Tomizuka Lei Li Zehuan Yuan Changhu Wang Ping Luo
references.title: Sparse r-cnn: End-to-end object detection with learnable proposals
references.venue: arXiv preprint arXiv:2011.12450,
references.authors: Mingxing Tan Ruoming Pang Quoc V. Le
references.title: EfficientDet: Scalable and efficient object detection
references.venue: In CVPR,
references.authors: Zhi Tian Chunhua Shen Hao Chen Tong He
references.title: FCOS: Fully convolutional one-stage object detection
references.venue: In ICCV,
references.authors: Yue Wu Yinpeng Chen Lijuan Wang Yuancheng Ye Zicheng Liu Yandong Guo Yun Fu
references.title: Large scale incremental learning
references.authors: Yongqin Xian Bernt Schiele Zeynep Akata
references.title: Zero-shot learning - the good, the bad and the ugly
references.authors: Shuo Yang Wei Yu Ying Zheng Hongxun Yao Tao Mei
references.title: Adaptive semantic-visual tree for hierarchical embeddings
references.venue: ACM MM,
references.authors: Shuo Yang Lu Liu Min Xu
references.title: Free lunch for few-shot learning: Distribution calibration
references.venue: In ICLR,
references.authors: Shuo Yang Min Xu Haozhe Xie Stuart Perry Jiahao Xia
references.title: Single-view 3d object reconstruction from shape priors in memory
references.venue: In CVPR,
references.authors: Alireza Zareian Kevin Dela Rosa Derek Hao Hu Shih-Fu Chang
references.title: Open-vocabulary object detection using captions
references.venue: In CVPR,
references.authors: Hongkai Zhang Hong Chang Bingpeng Ma Shiguang Shan Xilin Chen
references.title: Cascade RetinaNet: Maintaining consistency for single-stage object detection
references.venue: In BMVC,
references.authors: Xiaosong Zhang Fang Wan Chang Liu Rongrong Ji Qixiang Ye
references.title: FreeAnchor: Learning to match anchors for visual object detection
references.venue: NeurIPS,
references.authors: Xingyi Zhou Dequan Wang Philipp Krähenbühl
references.title: Objects as points
references.venue: arXiv preprint arXiv:1904.07850,
references.authors: Xingyi Zhou Jiacheng Zhuo Philipp Krahenbuhl
references.title: Bottom-up object detection by grouping extreme and center points
references.venue: In CVPR,
sections.heading: 1 INTRODUCTION
sections.text: Object detection, which aims at localizing and classifying objects in a given scene (Felzenszwalb et al., 2010; Everingham et al., 2010; Lin et al., 2014), is one of the most iconic abilities of biological intelligence. It was introduced to the artificial intelligence field to endow an intelligence agent with the ability of scene understanding. Although significant advances have been made to improve the object detection system in recent years (Girshick et al., 2014; Ren et al., 2015; Cai & Vasconcelos, 2018; Sun et al., 2020b; Redmon et al., 2016; Lin et al., 2017; Tian et al., 2019; Zhou et al., 2019a; Carion et al., 2020; Sun et al., 2020a), a strong assumption that all the objects of interest have been annotated in the training set, i.e., close-set learning, is always made but not holds well. All unknown objects are treated as background and ignored by current detectors, making such detectors cannot handle corner cases in many real-world applications such as autonomous driving, where some obstacles are not available during training, but must be detected when intelligent cars are running on the road. However, creating a large-scale dataset that contains annotations for all objects of interest at once is extremely expensive, even impossible.
  Superior to current detectors, humans naturally have the ability to discover both known and unknown objects and gradually learn novel concepts motivated by their curiosity. Learning by discovering the unknown is crucial for human intelligence (Livio, 2017; Meacham, 1983), and has been considered as a key step to achieve artificial general intelligence (AGI) (Goertzel, 2014). Recently, a new object detection paradigm, named Open-World Object Detection, has been established (Joseph et al., 2021; Miller et al., 2021; 2018c; Liu et al., 2020) to mimic this learning procedure.
  A qualified open-world object detector can not only identify known objects, but also discover object instances of unknown categories and gradually learn to recognize them when their annotations
  ∗corresponding author
  progressively arrive. The learning of novel categories is always in an incremental fashion, where the detector cannot access all old data when training on new categories. This open-world learning setting is much more realistic but challenging than previous close-set object detection.
  The open-world object detection poses two challenges on current detectors, i.e., recognition of unknown categories and incremental learning. On the one hand, previous close-set detectors do not explicitly encourage intra-class compactness (Liu et al., 2016b; Yang et al., 2021b). However, compact feature representation is highly required for unknown object recognition (discovery). If the known categories occupy most of the feature space, the detector will probably classify the unknown object as one of the known categories. On the other hand, the vanilla training strategy of the object detector lacks the mechanism to prevent ‘catastrophic forgetting’ in incremental learning (Joseph et al., 2021; Shmelkov et al., 2017a; Peng et al., 2020), i.e., previously-known objects features are severely distorted when learning new categories. Consequently, training on novel categories weakens the detector’s ability to detect previously-known objects.
  Previous works have made efforts to endow the object detector with the capacity of unknown recognition and incremental learning. One of the representative works, ORE (Joseph et al., 2021), designs a clustering loss function to compact the object features, and involves an energy-based outof-distribution identification approach (Liu et al., 2020) to detect unknown objects. Combining these two independent technologies in a step-wise manner makes ORE a non-end-to-end framework. Such solutions for open-world object detection are far from the optimal status. Additionally, ORE (Joseph et al., 2021) doesn’t guarantee a feature space topology consistency, which is crucial for effective new class learning and avoidance of catastrophic forgetting, as shown in Figure 1.
  This paper formalizes unknown recognition and incremental learning in a unified perspective and proposes a much simpler but more effective framework for open-world object detection than prior arts. We propose that an open-world object detector is desired to learn a feature space with such characteristics, including (a) discriminativeness: the unknown objects and the objects of new categories couldn’t be overlapped with any previously-known categories in the feature space, and (b) consistency: the feature topology of previously-known categories couldn’t be distorted severely when
  learning new categories. Our key idea is to pre-define a unique and fixed centroid in feature space for each category, including the ‘unknown’ category, and to push object instances close to their belonging centroids during the life-long learning of an open-world object detector. The pre-defined centroids are named as ‘Semantic Anchors’, and all semantic anchors constitute the structure of ‘Semantic Topology’. As shown in Figure 2, all features are binded to their corresponding semantic anchors to satisfy the discriminativeness, previously-known objects feature topology is maintained when incrementally learning new categories to satisfy the consistency.
  We introduce an off-the-shelf pre-trained language model to set up the semantic topology. The semantic anchor for each category is derived from the language model by embedding the corresponding category name. By incrementally registering new semantic anchors when new classes are involved, the semantic topology gradually grows. In the experiments, we show that by combining a current detector with our proposed semantic anchor head, a huge improvement in open-world object detection performance across the board is achieved. In addition to the consistent mAP improvement over the whole life of the detector, the ability of unknown recognition of our proposed method outperforms the current state-of-the-art methods by a large margin, e.g., the absolute open-set error is reduced by 2/3, from 7832 to 2546.
  More importantly, we conduct a comparison experiment by randomly generating semantic anchors instead of leveraging language models. Although randomly-generated anchors do not provide semantic priors, their performance still surpasses state-of-the-art methods. This strongly proves that the topology consistency is the most key characteristic for open-world learning while introducing semantic relationships can further boost the performance.
sections.heading: 2 RELATED WORKS
sections.text: Object detection
  Modern object detection frameworks (Ren et al., 2015; Redmon et al., 2016; Liu et al., 2016a; Lin et al., 2017; Zhang et al., 2019a; Redmon et al., 2016; Redmon & Farhadi, 2018; Girshick, 2015; Duan et al., 2019; Tian et al., 2019; Tan et al., 2020; Zhou et al., 2019b; Jiang et al., 2018; Zhang et al., 2019b; Carion et al., 2020; Sun et al., 2020b) take advantage of the high-capacity representation in deep neural networks to localize and classify the target class object in given images and videos. These well-established detectors can achieve excellent performance in the close-set dataset such as PascalVOC (Everingham et al., 2010), MSCOCO (Lin et al., 2014). However, the detectors can
  not handle open-world object detection, which is more common in the real world. To this end, ORE (Joseph et al., 2021) raises and formalizes the open-world object detection problem.
  Class incremental learning Class incremental learning aims to learn a classifier incrementally to recognize all encountered classes met so far, including previously-known classes and novel classes. Knowledge distillation is commonly adopted to mitigate forgetting old classes, which stores some old class exemplars to fine-tune the model or compute the distillation loss. iCaRL (Rebuffi et al., 2017) maintains an ‘episodic memory’ of the exemplars and incrementally adds novel class examples into the memory. Then the nearest neighbor classifier can be obtained incrementally. LwF (Li & Hoiem, 2017) proposes a modified cross-entropy loss to preserve the knowledge in the previous task. BiC (Wu et al., 2019) points out the data imbalance between old classes and new classes causes the network’s prediction biased towards new classes. However, the existing class incremental methods cannot handle the open-world problem, where the classifier should identify those ‘unknown’ classes and incrementally learns to recognize them, but existing methods would recognize unknown objects as background.
  Open-set Learning In open-set learning, the knowledge contained in the training set is incomplete, i.e., the examples encountered during inference may belong to a category that does not appear in the training set. OpenMax (Bendale & Boult, 2016) uses a Weibull distribution to identify unknown instances in the feature space of deep networks. OLTR (Liu et al., 2019) proposes to tackle the open-set recognition problem in a data imbalance setting by deep metric learning. Besides the open-set classification, Dhamija et al. (Dhamija et al., 2020b) found that the object detectors exhibit a high error rate that misclassifies unknown classes to known classes with high confidence. To solve this problem, many works (Miller et al., 2021; 2018a; Dhamija et al., 2020b) aims to measure the uncertainty of the detectors’ outputs to reject open-set error. Miller et al. (Miller et al., 2018a) uses Monte Carlo Dropout sampling to estimate the uncertainty in an SSD detector. After that, Miller et al. proposes to model a Gaussian Mixture distribution for each class in the detector’s feature space to reject unknown classes. ORE (Joseph et al., 2021) uses an energy-based out-of-distribution recognition method (Liu et al., 2020) to distinguish the known and unknown. However, such method needs to compute the energy score distribution among all instances, including known and unknown, during evaluation, making ORE a non-end-to-end method. We formulate the open-set recognition problem and the class incremental learning into a unified framework. Different from (Joseph et al., 2021), our method doesn’t access any unknown instances but achieves much superior performance.
  Zero-shot Learning Aligning image and text into a common feature space has always been an active research topic (Frome et al., 2013; Joulin et al., 2016; Li et al., 2017; Desai & Johnson, 2021; Yang et al., 2019; 2021a), especially in zero-shot learning (Xian et al., 2017; Bansal et al., 2018; Zareian et al., 2021; Gu et al., 2021). Many researches in zero-shot learning leverage the information in language models to assist zero-shot image classification (Xian et al., 2017) or zero-shot object detection (Bansal et al., 2018; Zareian et al., 2021; Gu et al., 2021). However, this paper tackles a different problem and has a different usage of language models. We identify that a consistent feature manifold topology plays an essential role in open-world object detection, and the language model is used for generating growth-able and consistent semantic topology to constrain the feature space learning of an open-world detection. Also, this paper is the first one to incorporate a language model, e.g., CLIP, to assist open-world object detection, which results in a simple training framework and strong empirical performance.
sections.heading: 3 METHODOLOGY
sections.text: In this section, we introduce the Open World Object Detection problem definition in Section 3.1, the method overview in Section 3.2, and the details of the proposed method in Section 3.3 and Section 3.4.
sections.heading: 3.1 PROBLEM DEFINITION
sections.text: An open world object detector should detect all the previously seen object classes, and can also identify if a testing instance known or unknown (belongs to previously seen classes or not). If unknown, the detector should gradually learn the unknown classes when their annotations progressively arrive without retraining from scratch. Formally, at each time point t, we assume there exists a set of known object classes Ctkn = {l1, l2, . . . , lC} and a set of unknown object classes Ctunk = {lC+1, lC+2, . . . }. The detector Dt at the time point t has only trained on classes in Ctkn while may encounter all classes
  including Ctkn and Ctukn during evaluation. Besides correctly classifying an object instance from the known classes Ctkn, the detector Dt should also label all instances from the unknown class set Ctukn as unknown. At the time point t+ 1, the unknown instances will be forwarded to a human user who can select n novel classes of interest to annotate and return them back to the model. The detector Dt should incrementally learn these n novel classes and updates itself to Dt+1 without retraining from scratch on the whole dataset. The known class set Ct+1kn in the time point t+ 1 is also updated to Ct+1kn = Ctkn + {lC+1, . . . , lC+n}. The data instances in the known classes Ckn are assumed to be labeled in the form of {x, y}, where x indicates the image and y indicates the annotations including class label l and object coordinates, i.e., y = [l, x, y, w, h] where x, y, w, h denote the bounding box center coordinates, width and height respectively.
sections.heading: 3.2 METHOD OVERVIEW
sections.text: A Region Proposal Network (RPN) that can identify unknown and a discriminative and consistent feature space are two critical components for an open world detector. Here, we adopt the UnknownAware RPN proposed in (Joseph et al., 2021) and propose to constrain the detector’s feature space topology with a pre-defined Semantic Topology. Specifically, we create a unique and fixed centroid for each category in the feature space, named semantic anchor. The semantic anchors for all classes are derived by feeding forward their class names into a pre-trained language model. Our key idea is to manipulate the detector’s feature space to be consistent with the semantic topology constituted by semantic anchors during the whole life of the detector. Due to the feature dimension discrepancy, a fully connected layer (semantic projector) is used to align the dimension between RoI features and semantic anchors. At the training stage, the semantic features outputted by the semantic projector are forced to cluster around their corresponding semantic anchors by a designed ‘SA (semantic anchor) Head’. When incremental learning, the SA Head gradually registers new semantic anchors for novel classes and continually pulls close the novel class features and their semantic anchors. To mitigate ‘catastrophic forgetting’ caused by old class feature distortion, the SA Head also minimizes the distance between some stored old class exemplars and their semantic anchors when learning novel classes. To better leverage the well-constructed feature space, we attach an additional classification layer to classify the semantic features. Figure 3 shows the training pipeline of the proposed openworld object detector. At inference, we multiply the class posterior probabilities produced by the two classification heads and get the final prediction.
sections.heading: 3.3 UNKNOWN-AWARE RPN
sections.text: Open-world object detectors are required to separate potential unknown objects from the background. Therefore, we need some specific designs for the Region Proposal Network (RPN). In this paper, we adopt the unknown-ware RPN proposed in (Joseph et al., 2021) which selects the top-k background region proposals, sorted by its objectness scores, as unknown objects. The unknown-aware RPN relies on the fact that Region Proposal Network is class agnostic. Given an input image, the RPN generates bounding box predictions for foreground and background instances, along with the corresponding objectness scores. The unknown-aware RPN labels those proposals with a high objectness score but do not overlap with any ground-truth object as a potential unknown object.
sections.heading: 3.4 SEMANTIC TOPOLOGY
sections.text: We propose to pre-define a semantic topology for detectors’ feature space rather than learn from data. The semantic topology is constituted by semantic anchors. Each semantic anchor is a pre-defined feature centroid for an object class. The semantic anchors can be generated by embedding corresponding class names using a pre-trained language model. The semantic topology can dynamically grow as novel classes are introduced to the open-world detector.
sections.heading: 3.4.1 SEMANTIC ANCHOR REGISTRATION
sections.text: We generate semantic anchors for all classes by feeding the class names into an off-the-shelf pretrained language model. Denote li ∈ Ctkn as class name of the i-th known class at time t andM as an off-the-shelf pre-trained language model. The semantic anchor for class li is defined as Ai =M(li), where Ai ∈ Rn, the dimension n depends on the pre-trained language model. The semantic anchor registration is performed repeatedly as long as known class set update Ctkn → C t+1 kn when novel classes are introduced at time t+ 1. Note we also register a semantic anchor for all instances labeled
  as unknown by the unknown-aware RPN to better distinguish the unknown instances from the known instances. Follows the same strategy, the semantic anchor for class unknown is also generated by the word embedding of unknown text.
sections.heading: 3.4.2 OBJECTIVE FUNCTION
sections.text: The RoI features f ∈ Rd are feature vectors generated by an intermediate layer of the object detector, which are used for category classification and bounding box regression. We manipulate the RoI features f to construct the detector’s feature manifold topology. Denote fi as an RoI feature of the i-th known class, we first align the dimension of fi as the dimension of its corresponding semantic anchor Ai, using a fully connected layer with d × n dimensional weights. The corresponding semantic feature is denoted as f̂i ∈ Rn. We constrain the detector’s feature manifold topology by clustering the semantic features around their corresponding semantic anchors, the learning objective is formalized as Lsa = ‖f̂i −Ai‖. (1) Minimizing this loss would ensure the desired feature space topology. To better leverage the constructed feature space, we use an additional classification head to classify the semantic features f̂i, with the same label space as RoI classification head. The total training objective is the combination of semantic anchor loss Lsa, semantic feature classification loss Lclsse , RoI feature classification loss and bounding box regression loss:
  Ltotal = Lsa + Lclsse + Lclsroi + Lreg. (2)
  At the inference stage, the classification results are computed by multiplying the two class posterior probability vectors predicted by the RoI feature classification head and the semantic feature classification head.
sections.heading: 3.4.3 TOPOLOGY STABILIZATION
sections.text: To store the detection ability on old classes, a balanced set of exemplars are stored and used to fine-tune the model after each incremental learning session as in the previous open-world object detection approach (Joseph et al., 2021). However, we argue that finetuning the detector ignoring the old knowledge topology as in (Joseph et al., 2021) still suffers from the severe ‘catastrophic forgetting’ problem. Benefitting from pre-defining the feature space topology, our method can guarantee a consistent feature space during the finetuning stage. The stored old class and new class instances are still forced to cluster around their pre-defined centroids to guarantee the feature space topology unchanged.
sections.heading: 4 EXPERIMENTS
sections.text: We introduce the evaluation protocol, including datasets and evaluation metrics, implementation details, and experimental results in this section.
sections.heading: 4.1 DATASETS
sections.text: Following (Joseph et al., 2021), the open-world detector is evaluated on all 80 object classes from Pascal VOC (Everingham et al., 2010) (20 classes) and MS-COCO (Lin et al., 2014) (20+60 classes). All categories are grouped into a set of tasks T = {T1, . . . , Tt, . . . }, where all categories of Tt will be introduced to the detector at a time point t. At the time point t, all categories from {Tτ |τ <= t} will be treated as known and {Tτ |τ > t} would be treated as unknown. As in (Joseph et al., 2021), T1 consists of all VOC classes, and the remaining 60 classes from MS-COCO are grouped into three successive tasks with semantic drifts. The open-world object detector is trained on the training set of all classes from Pascal VOC and MS-COCO, and evaluated on the Pascal VOC test split and MS-COCO val split. The validation set consists of 1k images from the training data of each task.
sections.heading: 4.2 EVALUATION METRICS
sections.text: We introduce three metrics to evaluate the detection performance of an open-world object detector on known classes and unknown classes at each time point:
  • Mean Average Precision (mAP) (Everingham et al., 2010; Lin et al., 2014). Following previous works (Joseph et al., 2021), we use mAP at IoU threshold of 0.5 to measure the performance on known classes. Since the object classes are continually involved, mAP is calculated on previouslyknown classes and currently-known classes.
  • Absolute Open-Set Error (A-OSE) (Joseph et al., 2021; Miller et al., 2018b). A-OSE counts the absolute number of unknown instances that are wrongly classified as any of the known classes. A-OSE evaluates the detector’s ability to avoid misclassifying unknown instances as one of the known classes.
  • Wilderness Impact (WI) (Joseph et al., 2021; Dhamija et al., 2020a). The definition is WI = PK PK∪U
  − 1, where PK is the precision of the detector when evaluated on known classes and PK∪U refers to the precision when evaluated on all classes including known and unknown. The recall level R is set to be 0.8. WI evaluates the detector’s ability to successfully detect unknown objects and classify them as ’unknown’ class.
sections.heading: 4.3 IMPLEMENTATION DETAILS
sections.text: We extend the traditional Faster R-CNN (Ren et al., 2015) object detector with a ResNet-50 (He et al., 2016) to be an open-world object detector. The RoI feature is extracted from the last residual block in the RoI head, which has a 2048 dimension. The 2048-dim RoI feature is used for computing traditional bounding box regression loss and classification loss as the same as in many previous works (Joseph et al., 2021; Ren et al., 2015). The semantic projector is a fully-connected layer to align the dimension of RoI features with the semantic anchors. The dimension of semantic anchors depends on the choice of pre-trained language model, e.g., the semantic anchor is 512-dim when using CLIP text encoder (Radford et al., 2021b) to embed categories names. The semantic feature outputted by the semantic projector is used for calculating semantic anchor loss Lsa and semantic feature classification Lclsse loss. The Lsa and the Lclsse are added to the standard regression loss and RoI classification loss to jointly optimize the detector. For the topology stabilization, we store 100 instances per class as the same as in (Joseph et al., 2021). To enable the classifier to handle a variable number of classes at different time points, we assume there exists a maximum number of classes to be added and set the classification logits of unseen classes to a large negative value to make their contribution to softmax output negligible, following (Joseph & Balasubramanian, 2020; Rajasegaran et al., 2020; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018).
  To make a fair comparison, we report experimental results obtained by re-running the ORE (Joseph et al., 2021) official code1. All the hyper-parameters and optimizers are also controlled to be exactly the same for all methods.
sections.heading: 4.4 OPEN-WORLD OBJECT DETECTION
sections.text: Table 1 shows the open-world object detection performance of our proposed method and other baselines. At task 1, all methods are trained over all 20 Pascal-VOC classes. Twenty novel object classes are introduced gradually at the following task. WI and A-OSE are used to quantify how unknown instances are confused with known class objects after training on each task. WI and A-OSE are both the less the better. ORE (Joseph et al., 2021) used an extra clustering loss to separate class features, and an energy-based unknown identifier that learns shifted Weibull distributions from
  1Official repository: https://github.com/JosephKJ/OWOD
  ORE (T1) ORE (T4) Ours (T1) Ours (T4)
  all class data (including known and unknown) in the validation set to reject open-set error. Our method doesn’t access any unknown data instances from the validation set and achieves much superior performance on WI and A-OSE than Faster R-CNN and ORE. In task 3, our proposed method achieves 1/3 A-OSE compared to ORE (2120 v.s. 6900) and 1/4 A-OSE compared to Faster R-CNN (2120 v.s. 9622). Our method also reduces the Wilderness Impact (WI) by a large margin than baselines. This indicates our method has a much better ability on unknown detection. Furthermore, our method also achieves consistent mAP improvement over the whole life of the detector. At incremental learning session (task 2,3,4), the performance of Faster R-CNN on old classes deteriorate severely (55.35 → 6.09). Faster R-CNN + Finetuning, ORE, and our method store some old class instances and fine-tune on them after learning new classes to restore the old class performance. However, the Faster R-CNN + Finetuning and ORE don’t guarantee the feature manifold topology consistent. The superior performance on old classes also proves that guaranteeing the feature manifold topology is critical.
sections.heading: 4.5 INCREMENTAL OBJECT DETECTION
sections.text: To verify the effectiveness of our proposed method, we also conduct experiments on class incremental object detection (iOD), where the detectors only need to continually recognize novel object classes without unknown identification. We use the standard iOD evaluation protocol as (Joseph et al., 2021; Shmelkov et al., 2017b) to evaluate all methods, where group of classes (10, 5, and the last class) from PASCAL VOC 2007 (Everingham et al., 2010) are incrementally learned by a detector trained on the remaining set of classes. Benefits from the well-constructed feature space using semantic anchors, our proposed method performs favorably well on the incremental object detection (iOD) task against several state-of-the-art 2. Notably, our method exhibits strong ability to incrementally learn new classes while not forget old classes. This is because our detector’s feature space is constrained to be incrementally growth-able by assigning each object class a unique feature space location.
sections.heading: 4.6 ABLATION STUDY
sections.text: The effect of language model. We derive semantic topology from pre-trained language models. To explore the effect of the choice of semantic topology, we conduct ablation studies in this section. To be specific, we generate semantic anchors using two pre-trained language models, i.e., CLIPtext (Radford et al., 2021a) and BERT (Devlin et al., 2019). The dimensions of semantic anchor generated by CLIP-text and BERT are 512 and 768, respectively. The comparison results are shown in
  Table 3. These two language models obtain similar results, which both outperform ORE, the previous state-of-the-art method in open-world object detection. To further explore the importance of semantic priors, we also generate semantic anchors by randomly and uniformly sampling, which means we assume the total number of classes is known which is not applicable in practice. Surprisingly, our experiment shows that semantic anchors as random vectors still surpass state-of-the-art methods. This demonstrates that a consistent topology, even without semantic priors, could also largely benefit the task of open-world object detection, which strongly proves that the discriminative and the consistent are two key characteristics for open-world learning. However, the randomly assigned class centers would hinder the learning ability of networks. Instead, a pretrained language model can perfectly overcome these problem.
  The effect of semantic anchor loss. In this section, we explore the importance of unknown class anchor, semantic anchor clustering loss Lsa, semantic anchor classification loss Lclsse and the RoI feature classification loss Lclsroi . The experiments in Table 4 shows the semantic anchor clustering loss and the semantic anchor classification loss are both indispensable in identifying unknown objects.
sections.heading: 5 CONCLUSION
sections.text: Open world object detection raises two challenging problems on current detectors, unknown recognition and incremental learning. Different from previous methods which combine domain-dependent technologies to respectively solve these two problems, our work provides a unified perspective, i.e.semantic topology. In the framework of semantic topology, all object instances from the same category, including ‘unknown’ category, are assigned to their corresponding pre-defined centroids. Therefore, a discriminative and consistent feature representations are ensured during the whole life of an open-world object detector. Aided with our proposed semantic topology, a huge improvement in open-world object detection performance across the board is achieved.
sections.heading: 6 ACKNOWLEDGEMENT
sections.text: Ping Luo was supported by the General Research Fund of HK No.27208720 and the HKU-TCL Joint Research Center for Artificial Intelligence.