summaries: This submission addresses the problem of constructing a good initialized pretrained network for the object detection task. Compared to previous work, the authors argue that pretraining should be adapted to object detection by focusing on object-level features learning. Instead of pretraining on ImageNet, the authors pretrain the model by designing an object level contrastive learning framework. Experiments on COCO and PASCAL show that the proposed method achieves higher performance.  The paper aims to design a self-supervised pre-training method for the downstream object detection task.  The authors propose to perform self-supervised object-level representation learning. Specifically, this method uses selective search to obtain object proposals in unlabeled images and extract the object-level features as in Mask R-CNN. Clear improvements have been achieved on COCO object detection and instance segmentation with Mask R-CNN. This paper provides a self supervised method tailored for object detection (in the RCNN paradigm), inspired from BYOL.   The general idea is to use a non-parametric method (selective search) to generate some boxes, then apply a BYOL-style loss between ROI-Aligned features of each box coming from different augmentations of the image.   It differs from other SSL methods (BYOL, SWAV,...) in that it instead of pretraining solely the backbone, this methods ensures that all the components of the detector (FPN, RCNN head) are pre-trained.  It shows convincing results on COCO and pascal as well as some subsets, in a few shot setting. The code is made available as part of the supplementary, thus helping with reproducibility.  This paper proposes SoCo, a method for self-supervised representation learning for downstream object detection and instance segmentation tasks.  They argue for stronger alignment between downstream tasks and self-supervised pre-training. They demonstrate this in the case of object detection by (a) developing a representation learning scheme that encourages object level scale and translation invariance and (b) pre-training several components of the downstream detector such as FPN and R-CNN head unlike prior work which mostly just trains the ResNet backbone.  The former property will help their model detect objects at several locations and scales as is needed for object detection.  The latter property helps them get stronger alignment between the upstream and downstream setups. To this end they propose "scale aware assignment". The ideas is that, representations for small objects are taken from finer FPN pyramid levels and for bigger objects are taken from coarser FPN pyramid levels. This matches what happens in the actual FPN where anchors at finer pyramid levels are smaller and at coarser pyramid levels are bigger.  The overall recipe is as follows: Selective search, a hand-crafted region proposal technique, is used to suggest boxes on all images. K of these boxes are randomly selected in each training step per image and three views of that image are constructed using random resized crop and resize operations. FPN features are extracted using RoiAlign for these box locations using "scale aware assignment". A BYOL loss is applied between representations of the same box across the three views. As in BYOL an EMA teacher network and projection head are used.  They pretrain R50-C4 and R50-FPN  models upstream and fine-tune corresponding Mask-RCNN models downstream for MS COCO and Pascal VOC object detection and MS COCO instance segmentation. Low data regime is explored using 5% and 10% "Mini" COCO setups. ImageNet linear eval results are included in the appendix. Ablation experiments evaluate whether stronger alignment in terms of architecture between pre-training and fine-tuning were helpful and sensitivity w.r.t. hyper-parameters such as view size, momentum for EMA, batch-size etc.
abstractText: Image-level contrastive representation learning has proven to be highly effective as a generic model for transfer learning. Such generality for transfer learning, however, sacrifices specificity if we are interested in a certain downstream task. We argue that this could be sub-optimal and thus advocate a design principle which encourages alignment between the self-supervised pretext task and the downstream task. In this paper, we follow this principle with a pretraining method specifically designed for the task of object detection. We attain alignment in the following three aspects: 1) object-level representations are introduced via selective search bounding boxes as object proposals; 2) the pretraining network architecture incorporates the same dedicated modules used in the detection pipeline (e.g. FPN); 3) the pretraining is equipped with object detection properties such as object-level translation invariance and scale invariance. Our method, called Selective Object COntrastive learning (SoCo), achieves state-of-the-art results for transfer performance on COCO detection using a Mask R-CNN framework. Code is available at https://github.com/hologerry/SoCo.
authors.name: Fangyun Wei
authors.name: Yue Gao
authors.name: Zhirong Wu
authors.name: Han Hu
authors.name: Stephen Lin
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A largescale hierarchical image database
references.venue: IEEE conference on computer vision and pattern recognition,
references.authors: Zhirong Wu Yuanjun Xiong X Yu Stella Dahua Lin
references.title: Unsupervised feature learning via non-parametric instance discrimination
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre H Richemond Elena Buchatskaya Carl Doersch Bernardo Avila Pires Zhaohan Daniel Guo Mohammad Gheshlaghi Azar
references.title: Bootstrap your own latent: A new approach to self-supervised learning
references.venue: arXiv preprint arXiv:2006.07733,
references.authors: Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross Girshick
references.title: Momentum contrast for unsupervised visual representation learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Xinlei Chen Haoqi Fan Ross Girshick Kaiming He
references.title: Improved baselines with momentum contrastive learning
references.venue: arXiv preprint arXiv:2003.04297,
references.authors: Yonglong Tian Chen Sun Ben Poole Dilip Krishnan Cordelia Schmid Phillip Isola
references.title: What makes for good views for contrastive learning
references.venue: arXiv preprint arXiv:2005.10243,
references.authors: Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin
references.title: Unsupervised learning of visual features by contrasting cluster assignments
references.venue: arXiv preprint arXiv:2006.09882,
references.authors: Xinlei Chen Kaiming He
references.title: Exploring simple siamese representation learning
references.venue: arXiv preprint arXiv:2011.10566,
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A simple framework for contrastive learning of visual representations
references.venue: In International conference on machine learning,
references.authors: Zhenda Xie Yutong Lin Zheng Zhang Yue Cao Stephen Lin Han Hu
references.title: Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning
references.venue: arXiv preprint arXiv:2011.10043,
references.authors: Xinlong Wang Rufeng Zhang Chunhua Shen Tao Kong Lei Li
references.title: Dense contrastive learning for self-supervised visual pre-training
references.venue: arXiv preprint arXiv:2011.09157,
references.authors: Ceyuan Yang Zhirong Wu Bolei Zhou Stephen Lin
references.title: Instance localization for selfsupervised detection pretraining
references.venue: arXiv preprint arXiv:2102.08318,
references.authors: Olivier J Hénaff Skanda Koppula Jean-Baptiste Alayrac Aaron van den Oord Oriol Vinyals João Carreira
references.title: Efficient visual pretraining with contrastive detection
references.venue: arXiv preprint arXiv:2103.10957,
references.authors: Jasper RR Uijlings Koen EA Van De Sande Theo Gevers Arnold WM Smeulders
references.title: Selective search for object recognition
references.venue: International journal of computer vision,
references.authors: Tsung-Yi Lin Piotr Dollár Ross Girshick Kaiming He Bharath Hariharan Serge Belongie
references.title: Feature pyramid networks for object detection
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Shaoqing Ren Kaiming He Ross Girshick Jian Sun
references.title: Faster r-cnn: Towards real-time object detection with region proposal networks
references.venue: arXiv preprint arXiv:1506.01497,
references.authors: Kaiming He Georgia Gkioxari Piotr Dollár Ross Girshick
references.title: Mask r-cnn
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Pascal Vincent Hugo Larochelle Isabelle Lajoie Yoshua Bengio Pierre-Antoine Manzagol Léon Bottou
references.title: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion
references.venue: Journal of machine learning research,
references.authors: Ruslan Salakhutdinov Geoffrey Hinton
references.title: Deep boltzmann machines. In Artificial intelligence and statistics, pages 448–455
references.venue: PMLR,
references.authors: Gustav Larsson Michael Maire Gregory Shakhnarovich
references.title: Colorization as a proxy task for visual understanding
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Carl Doersch Abhinav Gupta Alexei A Efros
references.title: Unsupervised visual representation learning by context prediction
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Deepak Pathak Philipp Krahenbuhl Jeff Donahue Trevor Darrell Alexei A Efros
references.title: Context encoders: Feature learning by inpainting
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Spyros Gidaris Praveer Singh Nikos Komodakis
references.title: Unsupervised representation learning by predicting image rotations
references.venue: arXiv preprint arXiv:1803.07728,
references.authors: Olivier Henaff
references.title: Data-efficient image recognition with contrastive predictive coding
references.venue: In International Conference on Machine Learning,
references.authors: Jure Zbontar Li Jing Ishan Misra Yann LeCun Stéphane
references.title: Deny. Barlow twins: Selfsupervised learning via redundancy reduction
references.venue: arXiv preprint arXiv:2103.03230,
references.authors: Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin
references.title: Emerging properties in self-supervised vision transformers
references.venue: arXiv preprint arXiv:2104.14294,
references.authors: Mathilde Caron Piotr Bojanowski Julien Mairal Armand Joulin
references.title: Unsupervised pretraining of image features on non-curated data
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Priya Goyal Mathilde Caron Benjamin Lefaudeux Min Xu Pengchao Wang Vivek Pai Mannat Singh Vitaliy Liptchinsky Ishan Misra Armand Joulin
references.title: Self-supervised pretraining of visual features in the wild
references.venue: arXiv preprint arXiv:2103.01988,
references.authors: Pedro O Pinheiro Amjad Almahairi Ryan Y Benmaleck Florian Golemo Aaron Courville
references.title: Unsupervised learning of dense visual representations
references.venue: arXiv preprint arXiv:2011.05499,
references.authors: Pablo Arbeláez Jordi Pont-Tuset Jonathan T Barron Ferran Marques Jitendra Malik
references.title: Multiscale combinatorial grouping
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Zhigang Dai Bolun Cai Yugeng Lin Junying Chen
references.title: Up-detr: Unsupervised pre-training for object detection with transformers
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Songtao Liu Zeming Li Jian Sun
references.title: Self-emd: Self-supervised object detection without imagenet
references.venue: arXiv preprint arXiv:2011.13677,
references.authors: Bharat Singh Larry S. Davis
references.title: An analysis of scale invariance in object detection - snip, 2018
references.authors: Ross Girshick Jeff Donahue Trevor Darrell Jitendra Malik
references.title: Rich feature hierarchies for accurate object detection and semantic segmentation
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Ross Girshick
references.title: Fast r-cnn
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate shift
references.venue: In International conference on machine learning,
references.authors: Vinod Nair Geoffrey E Hinton
references.title: Rectified linear units improve restricted boltzmann machines
references.venue: In Icml,
references.authors: Yang You Igor Gitman Boris Ginsburg
references.title: Scaling sgd batch size to 32k for imagenet training
references.venue: arXiv preprint arXiv:1708.03888,
references.authors: Ilya Loshchilov Frank Hutter
references.title: Sgdr: Stochastic gradient descent with warm restarts
references.venue: arXiv preprint arXiv:1608.03983,
references.authors: Priya Goyal Piotr Dollár Ross Girshick Pieter Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia Kaiming He
references.title: Accurate, large minibatch sgd: Training imagenet in 1 hour
references.venue: arXiv preprint arXiv:1706.02677,
references.authors: Tsung-Yi Lin Michael Maire Serge Belongie James Hays Pietro Perona Deva Ramanan Piotr Dollár C Lawrence Zitnick
references.title: Microsoft coco: Common objects in context
references.venue: In European conference on computer vision,
references.authors: Mark Everingham Luc Van Gool Christopher KI Williams John Winn Andrew Zisserman
references.title: The pascal visual object classes (voc) challenge
references.venue: International journal of computer vision,
references.authors: Tete Xiao Colorado J Reed Xiaolong Wang Kurt Keutzer Trevor Darrell
references.title: Region similarity representation learning
references.venue: arXiv preprint arXiv:2103.12902,
references.authors: Agrim Gupta Piotr Dollar Ross Girshick
references.title: Lvis: A dataset for large vocabulary instance segmentation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Tsung-Yi Lin Priya Goyal Ross Girshick Kaiming He Piotr Dollár
references.title: Focal loss for dense object detection
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Zhi Tian Chunhua Shen Hao Chen Tong He
references.title: Fcos: Fully convolutional one-stage object detection
references.venue: In Proceedings of the IEEE/CVF international conference on computer vision,
references.authors: Kai Chen Jiaqi Wang Jiangmiao Pang Yuhang Cao Yu Xiong Xiaoxiao Li Shuyang Sun Wansen Feng Ziwei Liu Jiarui Xu
references.title: Mmdetection: Open mmlab detection toolbox and benchmark
references.venue: arXiv preprint arXiv:1906.07155,
sections.heading: 1 Introduction
sections.text: Pretraining and finetuning has been the dominant paradigm of training deep neural networks in computer vision. Downstream tasks usually leverage pretrained weights learned on large labeled datasets such as ImageNet [1] for initialization. As a result, supervised ImageNet pretraining has been prevalent throughout the field. Recently, self-supervised pretraining [2, 3, 4, 5, 6, 7, 8, 9] has achieved considerable progress and alleviated the dependency on labeled data. These methods aim to learn generic visual representations for various downstream tasks by means of image-level pretext tasks, such as instance discrimination. Some recent works [10, 11, 12, 13] observe that the image-level representations are sub-optimal for dense prediction tasks such as object detection and semantic segmentation. A potential reason is that image-level pretraining may overfit to holistic representations and fail to learn properties that are important outside of image classification.
  The goal of this work is to develop self-supervised pretraining that is aligned to object detection. In object detection, bounding boxes are widely adopted as the representation for objects. Translation and scale invariance for object detection are reflected by the location and size of the bounding boxes. An obvious representation gap exists between image-level pretraining and the object-level bounding boxes of object detection.
  Motivated by this, we present an object-level self-supervised pretraining framework, called Selective Object COntrastive learning (SoCo), specifically for the downstream task of object detection. To introduce object-level representations into pretraining, SoCo utilizes off-the-shelf selective search [14] to generate object proposals. Different from prior image-level contrastive learning methods which
  ∗Equal contribution.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  treat the whole image as an instance, SoCo treats each object proposal in the image as an independent instance. This enables us to design a new pretext task for learning object-level visual representations with properties that are compatible with object detection. Specifically, SoCo constructs object-level views where the scales and locations of the same object instance are augmented. Contrastive learning follows to maximize the similarity of the object across augmented views.
  The introduction of the object-level representation also allows us to further bridge the gap in network architecture between pretraining and finetuning. Object detection often involves dedicated modules, e.g., feature pyramid network (FPN) [15], and special-purpose sub-networks, e.g., R-CNN head [16, 17]. In contrast to image-level contrastive learning methods where only feature backbones are pretrained and transferred, SoCo performs pretraining over all the network modules used in detectors. As a result, all layers of the detectors can be well-initialized.
  Experimentally, the proposed SoCo achieves state-of-the-art transfer performance from ImageNet to COCO. Concretely, by using Mask R-CNN with an R50-FPN backbone, it obtains 43.2 APbb / 38.4 APmk on COCO with a 1× schedule, which are +4.3 APbb / +3.0 APmk better than the supervised pretraining baseline, and 44.3 APbb / 39.6 APmk on COCO with a 2× schedule, which are +3.0 APbb / +2.3 APmk better than the supervised pretraining baseline. When transferred to Mask R-CNN with an R50-C4 backbone, it achieves 40.9 APbb / 35.3 APmk and 42.0 APbb / 36.3 APmk on the 1× and 2× schedule, respectively.
sections.heading: 2 Related Work
sections.text: Unsupervised feature learning has a long history in training deep neural networks. Auto-encoders [18] and Deep Boltzmann Machines [19] learn layers of representations by reconstructing image pixels. Unsupervised pretraining is often used as a method for initialization, which is followed by supervised training on the same dataset such as handwritten digits for classification.
  Recent advances in unsupervised learning, especially self-supervised learning, tend to formulate the problem in a transfer learning scenario, where pretraining and finetuning are trained on different datasets with different purposes. Pretext tasks such as colorization [20], context prediction [21], inpainting [22] and rotation prediction [23] force the network to learn semantic information in order to solve the pretext tasks. Contrastive methods based on the instance discrimination pretext task [2] learn to map augmented views of the same instance to similar embeddings. Such approaches [24, 4, 5, 9, 25, 26] have shown strong transfer ability for a number of downstream tasks, sometimes even outperforming the supervised counterpart by large margins [4].
  With new technical innovations such as learning clustering assignments [7], large-scale contrastive learning has been successfully applied on non-curated data sets [27, 28]. While the linear evaluation result on ImageNet classification has improved significantly [3], the progress on transfer performance for dense prediction tasks has been limited. Due to this, a growing number of works investigate pretraining specifically for object detection and semantic segmentation. The idea is to shift imagelevel representations to pixel-level or region-level representations. VADer [29], PixPro [10] and DenseCL [11] propose to learn pixel-level representations by matching point features of the same physical location under different views. InsLoc [12] learns to match region-level features from composited imagery. DetCon [13] additionally uses the bottom-up segmentation of MCG [30] for supervising intra-segment pixel variations. UP-DETR [31] proposes a pretext task named random query patch detection to pretrain DETR detector. Self-EMD [32] explores self-supervised representation learning for object detection without using Imagenet images.
  Our work is also motivated and inspired by object detection methods which advocate architectures and training schemes invariant to translation and scale transformations [15, 33]. We find that incorporating them in the pretraining stage also benefits object detection, which has largely been overlooked in previous self-supervised learning works.
sections.heading: 3 Method
sections.text: We introduce a new contrastive learning method which maximizes the similarity of object-level features representing different augmentations of the same object. Meanwhile, we introduce architectural alignment and important properties of object detection into pretraining when designing the objectlevel pretext task. We use an example where transfer learning is performed on Mask-RCNN with a R50-FPN backbone to illustrate the core design principles. To further demonstrate the extensibility
  and flexibility of our method, we also apply SoCo on a R50-C4 structure. In this section, we first present an overview of the proposed SoCo in Section 3.1. Then we describe the process of object proposal generation and view construction in Section 3.2. Finally, the object-level contrastive learning and our design principles are introduced in Section 3.3.
sections.heading: 3.1 Overview
sections.text: Figure 1 displays an overview of SoCo. SoCo aims to align pretraining to object detection in two aspects: 1) network architecture alignment between pretraining and object detection; 2) introducing central properties of detection. Concretely, besides pretraining a backbone as done in existing selfsupervised contrastive learning methods, SoCo also pretrains all the network modules used in an object detector, such as FPN and the head in the Mask R-CNN framework. As a result, all layers of the detector can be well-initialized. Furthermore, SoCo strives to learn object-level representations which are not only meaningful, but also invariant to translation and scale. To achieve this, it encourages diversity of scales and locations of objects by constructing multiple augmented views and applying a scale-aware assignment strategy for different levels of a feature pyramid. Finally, object-level contrastive learning is applied to maximize the feature similarity of the same object across augmented views.
sections.heading: 3.2 Data Preprocessing
sections.text: Object Proposal Generation. Inspired by R-CNN [34] and Fast R-CNN [35], we use selective search [14], an unsupervised object proposal generation algorithm which takes into account color similarity, texture similarity, size of region and fit between regions, to generate a set of object proposals for each of the raw images. We represent each object proposal as a bounding box b = {x, y, w, h}, where (x, y) denotes the coordinates of the bounding box center, and w and h are the corresponding width and height, respectively. We keep only the proposals that satisfy the following requirements: 1) 1/3 ≤ w/h ≤ 3; 2) 0.3 ≤ √ wh/ √ WH ≤ 0.8, where W and H denote width and height of the input image. The object proposal generation step is performed offline. In each training iteration, we randomly select K proposals for each input image.
  View Construction. Three views, namely V1, V2 and V3, are used in SoCo. The input image is resized to 224× 224 to obtain V1. Then we apply a random crop with a scale range of [0.5, 1.0] on V1 in generating V2. V2 is then resized to the same size as V1 and object proposals outside of V2 are dropped. Next, we downsample V2 to a fixed size (e.g. 112 × 112) to produce V3. In all of these cases, the bounding boxes transformed according to the cropping and resizing of the RGB images
  (see Figure 1, Data Preprocessing). Finally, each view is randomly and independently augmented. We adopt the augmentation pipeline of BYOL [3] but discard the random crop augmentation since spatial transformation is already applied on all three views. Notice that the scale and location of the same object proposal are different across the augmented views, which enables the model to learn translation-invariant and scale-invariant object-level representations.
  Box Jitter. To further encourage variance of scales and locations of object proposals across views, we adopt a box jitter strategy on the generated proposals as an object-level data augmentation. Specifically, given an object proposal b = {x, y, w, h}, we randomly generate a jittered box b̂ = {x̂, ŷ, ŵ, ĥ} as follows: 1) x̂ = x + r · w; 2) ŷ = y + r · h; 3) ŵ = w + r · w; 4) ĥ = h + r · h, where r ∈ [−0.1, 0.1]. The box jitter is randomly applied on each proposal with a probability of 0.5.
sections.heading: 3.3 Object-Level Contrastive Learning
sections.text: The goal of SoCo is to align pretraining to object detection. Here, we use the representative framework Mask R-CNN [17] with feature pyramid network (FPN) [15] to demonstrate our key design principles. The alignment mainly involves aligning the pretraining architecture with that of object detection and integrating important object detection properties such as object-level translation invariance and scale invariance into the pretraining.
  Aligning Pretraining Architecture to Object Detection. Following Mask R-CNN, we use a backbone with FPN as the image-level feature extractor f I . We denote the output of FPN as {P2, P3, P4, P5} with a stride of {4, 8, 16, 32}. Here, we do not use P6 due to its low resolution. With the bounding box representation b, RoIAlign [17] is applied to extract the foreground feature from the corresponding scale level. For further architectural alignment, we additionally introduce an R-CNN head fH into pretraining. The object-level feature representation h of bounding box b is extracted from an image view V as:
  h = fH(RoIAlign(f I(V ), b)). (1)
  SoCo uses two neural networks to learn, namely an online network and a target network. The online network and the target network share the same architecture but with different sets of weights. Concretely, the target network weights f Iξ , f H ξ are the exponential moving average (EMA) with the momentum coefficient τ of the online parameters f Iθ and f H θ . Denote the set of object proposals {bi} considered in the image. Let hi be the object-level representation of proposal bi in view V1, and h′i, h ′′ i be the representation of bi in view V2, V3. They are extracted using the online network and the target network respectively,
  hi = f H θ (RoIAlign(f I θ (V1), bi)), (2)
  h′i = f H ξ (RoIAlign(f I ξ (V2), bi)), h ′′ i = f H ξ (RoIAlign(f I ξ (V3), bi)). (3)
  We follow BYOL [3] for learning contrastive representations. The online network is appended with a projector gθ, and a predictor qθ for obtaining latent embeddings. Both gθ and qθ are two-layer MLPs. The target network is only appended with the projector gξ for avoiding trivial solutions. We use vi, v′i and v′′i to denote the latent embeddings of object-level representations {hi, h ′ i, h ′′ i }, respectively:
  vi = qθ(gθ(hi)), v ′ i = gξ(h ′ i), v ′′ i = gξ(h ′′ i ). (4)
  The contrastive loss for the i-th object proposal is defined as:
  Li = −2 · 〈vi, v′i〉 ‖vi‖2 · ‖v′i‖2 − 2 · 〈vi, v ′′ i 〉 ‖vi‖2 · ‖v′′i ‖2 . (5)
  Then we can formulate the overall loss function for each image as:
  L = 1 K K∑ i=1 Li, (6)
  where K is the number of object proposals. We symmetrize the loss L in Eq. 6 by separately feeding V1 to the target network and {V2, V3} to the online network to compute L̃. At each training iteration, we perform a stochastic optimization step to minimize LSoCo = L+ L̃.
  Scale-Aware Assignment. Mask R-CNN with FPN uses the IoU between anchors and ground-truth boxes to determine positive samples. It defines anchors to have areas of {322, 642, 1282, 2562} pixels on {P2, P3, P4, P5}, respectively, which means ground-truth boxes of scale within a range are assigned to a specific pyramid level. Inspired by this, we propose a scale-aware assignment strategy, which greatly encourages the pretraining model to learn object-level scaleinvariant representations. Concretely, we assign object proposals of area within a range of {0 − 482, 492 − 962, 972 − 1922, 1932 − 2242} pixels to {P2, P3, P4, P5}, respectively. Notice that the maximum proposal size is 224× 224 since all of the views are resized to a fixed resolution smaller than 224. The advantage is that the same object proposals at different scales are encouraged to learn consistent representations through contrastive learning. As a result, SoCo learns object-level scale-invariant visual representations, which is important for object detection.
  Introducing Properties of Detection to Pretraining. Here, we discuss how SoCo promotes important properties of object detection in the pretraining. Object detection uses tight bounding boxes to represent objects. To introduce object-level representations, SoCo generates object proposals by selective search. Translation invariance and scale invariance at the object level are regarded as the most important properties for object detection, i.e., feature representations of objects belonging to same category should be insensitive to scale and location. Recall that V2 is a randomly cropped patch of V1. Random cropping introduces box shift and thus contrastive learning between V1 and V2 encourages the pretraining model to learn location-invariant representations. V3 is generated by downsampling V2, which results in a scale augmentation of object proposals. With our scale-aware assignment strategy, the contrastive loss between V1 and V3 guides the pretraining towards learning scale-invariant visual representations.
  Extending to Other Object Detectors. SoCo can be easily extended to align to other detectors besides Mask R-CNN with FPN. Here, we apply SoCo to Mask R-CNN with a C4 structure, which is a popular non-FPN detection framework. The modification is three-fold: 1) for all object proposals, RoIAlign is performed on C4; 2) the R-CNN head is replaced by the entire 5-th residual block; 3) view V3 is discarded and the remaining V1 and V2 are kept for object-level contrastive learning. Experiments and comparisons with state-of-the-art methods in Section 4.3 demonstrate the extensibility and flexibility of SoCo.
sections.heading: 4 Experiments
sections.heading: 4.1 Pretraining Settings
sections.text: Architecture. Through the introduction of object proposals, the architectural discrepancy is reduced between pretraining and downstream detection finetuning. Mask R-CNN [17] is a commonly adopted framework to evaluate transfer performance. To demonstrate the extensibility and flexibility of SoCo, we provide details of SoCo alignment for the detection architectures R50-FPN and R50C4. SoCo-R50-FPN: ResNet-50 [36] with FPN [15] is used as the image-level feature encoder. RoIAlign [17] is then used to extract RoI features on feature maps {P2, P3, P4, P5} with a stride of {4, 8, 16, 32}. According to the image areas of object proposals, each RoI feature is then transformed to an object-level representation by the head network as in Mask R-CNN. SoCo-R50-C4: on the standard ResNet-50 architecture, we insert the RoI operation on the output of the 4-th residual block. The entire 5-th residual block is treated as the head network to encode object-level features. Both the projection network and prediction network are 2-layer MLPs which consist of a linear layer with output size 4096 followed by batch normalization [37], rectified linear units (ReLU) [38], and a final linear layer with output dimension 256.
  Dataset. We adopt the widely used ImageNet [1] which consists of ∼1.28 million images for self-supervised pretraining.
  Data Augmentation. Once all views are constructed, we employ the data augmentation pipeline of BYOL [3]. Specifically, we apply random horizontal flip, color distortion, Gaussian blur, grayscaling, and the solarization operation. We remove the random crop augmentation since spatial transformations have already been applied on all views.
  Optimization. We use a 100-epoch training schedule in all the ablation studies and report the results of 100-epochs and 400-epochs in the comparisons with state-of-the-art methods. We use the LARS optimizer [39] with a cosine decay learning rate schedule [40] and a warm-up period of 10 epochs.
  The base learning rate lrbase is set to 1.0 and is scaled linearly [41] with the batch size (lr = lrbase× BatchSize/256). The weight decay is set to 1.0 × e−5. The total batch size is set to 2048 over 16 Nvidia V100 GPUs. For the update of the target network, following [3], the momentum coefficient τ starts from 0.99 and is increased to 1 during training. Synchronized batch normalization is enabled.
sections.heading: 4.2 Transfer Learning Settings
sections.text: COCO [42] and Pascal VOC [43] datasets are used for transfer learning. Detectron2 [44] is used as the code base.
  COCO Object Detection and Instance Segmentation. We use the COCO train2017 set which contains ∼118k images with bounding box and instance segmentation annotations in 80 object categories. Transfer performance is evaluated on the COCO val2017 set. We adopt the Mask R-CNN detector [17] with the R50-FPN and the R50-C4 backbones. We report APbb, APbb50 and AP bb 75 for object detection, and APmk, APmk50 and AP mk 75 for instance segmentation. We report the results under the COCO 1× and 2× schedules in comparisons with state-of-the-art methods. All ablation studies are conducted on the COCO 1× schedule. Pascal VOC Object Detection. We use the Pascal VOC trainval07+12 set which contains∼16.5k images with bounding box annotations in 20 object categories as the training set. Transfer performance is evaluated on the Pascal VOC test2007 set. We report the results of APbb, APbb50 and AP bb 75 for object detection.
  Optimization. All the pretrained weights except for projection and prediction are loaded into the object detection network for transfer. Following [5], synchronized batch normalization is used across all layers including the newly initialized batch normalization layers in finetuning. For both the COCO and Pascal VOC datasets, we finetune with stochastic gradient descent and a batch size of 16 split across 8 GPUs. For COCO transfer, we use a weight decay of 2.5× e−5, and a base learning rate of 0.02 that increases linearly for the first 1000 iterations and drops twice by a factor of 10, after 23 and 8 9 of the total training time. For Pascal VOC transfer, the weight decay is 1.0 × e
  −4, and the base learning rate is set to 0.02 and divided by 10 at 34 and 11 12 of the total training time.
sections.heading: 4.3 Comparison with State-of-the-Art Methods
sections.text: Mask R-CNN with R50-FPN on COCO. Table 1 shows the transfer results for Mask R-CNN with R50-FPN backbone. We compare SoCo with the state-of-the-art unsupervised pretraining methods on the COCO 1× and 2× schedules. We report our results under 100 epochs and 400 epochs of pretraining. On the 100-epoch training schedule, SoCo achieves 42.3 APbb / 37.6 APmk and 43.2 APbb / 38.8 APmk on the 1× and 2× schedules, respectively. Pretrained for 400 epochs, SoCo outperforms all previous pretraining methods designed for either image classification or object detection, achieving 43.0 APbb / 38.2 APmk for the 1× schedule and 44.0 APbb / 39.0 APmk for the 2×
  schedule. Moreover, we propose an enhanced version named SoCo*, which constructs an additional view V4 for pretraining. Similar to the construction step of V2, V4 is a randomly cropped patch of V1. We resize V4 to 192× 192 and perform the same forward process as for V2 and V3. Compared with SoCo, SoCo* further boosts the performance and obtains an improvement of +0.2 APbb / +0.2 APmk on the 1× schedule, achieving 43.2 APbb / 38.4 APmk, and +0.3 APbb / +0.6 APmk on the 2× schedule, achieving 44.3 APbb / 39.6 APmk, respectively.
  Mask R-CNN with R50-C4 on COCO. To demonstrate the extensibility and flexibility of SoCo, we also evaluate transfer performance using Mask R-CNN with R50-C4 backbone on the COCO benchmark. We report the results of SoCo under 100 epochs and 400 epochs. Table 2 compares the proposed method to previous state-of-the-art methods. Without bells and whistles, SoCo obtains state-of-the-art performance, achieving 40.9 APbb / 35.3 APmk and 42.0 APbb / 36.3 APmk on the COCO 1× and 2× schedules, respectively. Faster R-CNN with R50-C4 on Pascal VOC. We also evaluate the transfer ability of SoCo on the Pascal VOC benchmark. Following [5], we adopt Faster R-CNN with R50-C4 backbone for transfer learning. Table 3 shows the comparison. SoCo obtains an improvement of +6.2 APbb against the supervised pretraining baseline, achieving 59.7 APbb for VOC object detection.
sections.heading: 4.4 Ablation Study
sections.text: To further understand the advantages of SoCo, we conduct a series of ablation studies that examine the effectiveness of object-level contrastive learning, the effects of alignment between pretraining and detection, and different hyper-parameters. For all ablation studies, we use Mask R-CNN with R50-FPN backbone for transfer learning and adopt a 100-epoch SoCo pretraining schedule. Transfer performance is evaluated under the COCO 1× schedule. For the ablation of each hyper-parameter or component, we fix all other hyper-parameters to the following default settings: view V3 with 112× 112 resolution, batch size of 2048, momentum coefficient τ = 0.99, proposal number K = 4,
  box jitter and scale-aware assignment are used, FPN and R-CNN head are pretrained and transferred, and selective search is used as the proposal generator.
  Effectiveness of Aligning Pretraining to Object Detection. We ablate each component of SoCo step by step to demonstrate the the effectiveness of aligning pretraining to object detection. Table 4 reports the studies. The baseline treats the whole image as an instance, without considering any detection properties or architecture alignment. It obtains 38.1 APbb and 34.4 APmk. Selective search introduces object-level representations. By leveraging generated object proposals and conducting contrastive learning at the object level, our method obtains an improvement of +2.5 APbb and +2.4 APmk. Next, we further minimize the architectural discrepancy between pretraining and object detection pipeline by introducing FPN and an R-CNN head into the pretraining architecture. We observe that only introducing FPN into pretraining slightly hurt the performance. In contrast, including both FPN and the R-CNN head improves the transfer performance to 41.2 APbb / 37.0 APmk, which verifies the effectiveness of architectural alignment between self-supervised pretraining and downstream tasks. On top of architectural alignment, we further leverage scale-aware assignment which encourages scale-invariant object-level visual representations, and an improvement of +3.5 APbb / +2.9 APmk against the baseline is found. The box jitter strategy slightly elevates performance. Finally, multiple views (V3) further directs SoCo towards learning scale-invariant and translationinvariant representations, and our method achieves 42.3 APbb and 37.6 APmk.
  Ablation Study on Hyper-Parameters. Table 5 examines sensitivity to the hyper-parameters of SoCo.
  Table 5(a) ablates the resolution of view V3. SoCo is found to be insensitive to the image size of V3. We use 112 × 112 as the default resolution due to its slightly better transfer performance and low training computation cost.
  Table 5(b) ablates the training batch size of SoCo. It can be seen that larger or smaller batch sizes hurt the performance. We use a batch size of 2048 by default.
  Table 5(c) ablates the effectiveness of object proposal generation strategies. To demonstrate the superiority of meaningful proposals generated by selective search, we randomly generate K bounding boxes in each of the training images to replace the selective search proposals (namely Random in Table 5(c)). SoCo can still yield satisfactory results using a single random bounding box to learn object-level representations, which is not surprising, since the RoIAlign operation performed on the randomly generated bounding boxes also introduces object-level representation learning. This fact indirectly demonstrates that downstream object detectors can benefit from self-supervised pretraining that involves object-level representations. However, the result is still slightly worse than the case where only one object proposal generated by selective search is used in the pretraining. For the cases where we randomly generate 4 and 8 bounding boxes, the noisy and meaningless proposals cause the pretraining to diverge. From the table, we also find that applying more object proposals (K = 8 and K = 16) generated by selective search can be harmful. One potential reason is that most of the images in the ImageNet dataset only contain a few objects, so a larger K may create duplicated and redundant proposals.
  Table 5(d) ablates the momentum coefficient τ of the exponential moving average, and τ = 0.99 yields the best performance.
sections.heading: 4.5 More Experiments
sections.text: Transfer Learning on LVIS Dataset. In addition to COCO and PASCAL VOC object detection, we also consider the challenging LVIS v1 dataset [46] to demonstrate the effectiveness and generality of our approach. The LVIS v1 detection benchmark contains 1203 categories in a long-tailed distribution with few training samples. It is considered to be more challenging than the COCO benchmark. We use Mask R-CNN with R50-FPN backbone and follow the standard LVIS 1× and 2× training schedules. We do not use any training or post-processing tricks. Table 6 shows the results. We can see that our method achieves +5.9 APbb/+5.6 APmk and +4.9 APbb/+4.6 APmk improvements under the LVIS 1× and 2× training schedules, respectively. Transfer Learning on Different Object Detectors. In addition to two-stage detectors, we further conduct experiments on RetinaNet [47] and FCOS [48], which are representative works for singlestage detectors and anchor-free detectors. We transfer the weights of backbone and FPN layers learned by SoCo* to RetinaNet and FCOS architectures and follow the standard COCO 1× schedule. We use MMDetection [49] as code base and default optimization configs are adopted. Table 7 shows the comparison.
  Evaluation on Mini COCO. Transfer to the full COCO dataset may be of limited significance due to the extensive supervision available from its large-scale annotated training data. To further demonstrate the generalization ability of SoCo, we conduct an experiment on a mini version of the COCO dataset, named Mini COCO. Concretely, we randomly select 5% or 10% of the training data from COCO train2017 to form Mini COCO benchmarks. Mask-RCNN with R50-FPN backbone is adopted to evaluate the transfer performance on the COCO 1× schedule. COCO val2017 is still used
  as the evaluation set. The other settings remain unchanged. Table 8 summarizes the results. Compared with the supervised pretraining baseline which achieves 19.4 APbb / 18.3 APmk and 24.7 APbb / 22.9 APmk on the 5% and 10% Mini COCO benchmarks, SoCo obtains large improvements of +6.6 APbb / +4.6 APmk and +5.7 APbb / +3.9 APmk, respectively. SoCo* further boosts performance with improvements of +7.4 APbb / +5.5 APmk and +6.4 APbb / +4.7 APmk over the supervised pretraining baseline.
  Pretraining on Non-object-centric Dataset. We use COCO training set and unlabeled set as pretraining data. Since ImageNet is ∼5.3 times larger than COCO, we pretrain our method on COCO for 100 epochs and prolonged 530 epochs respectively. Furthermore, we also generate a dataset named ImageNet-Subset which contains the same number of images of COCO to verify the impact of different pretraining datasets. Table 9 shows the results. Compared with the ImageNet pretrained model, the COCO pretrained model under 530 epochs is lower by 1.7 AP, possibly because the scale of COCO is smaller than ImageNet. Compared with the ImageNet-Subset pretrained model, the COCO pretrained model under 100 epochs is +0.4 AP higher, which demonstrates the importance of domain alignment between the pretraining dataset and detection dataset.
  Longer Finetuning. We transfer SoCo* to Mask R-CNN with R50-FPN under the COCO 4× schedule. Table 10 shows the results. Our model continues to improve the performance with the 4× finetuning schedule, surpassing the supervised baseline by +2.6 APbb/ +1.9 APmk.
sections.heading: 5 Conclusion
sections.text: In this paper, we propose a novel object-level self-supervised pretraining method named Selective Object COntrastive learning (SoCo), which aims to align pretraining to object detection. Different from prior image-level contrastive learning methods which treat the whole image as an instance, SoCo treats each object proposal generated by the selective search algorithm as an independent instance, enabling SoCo to learn object-level visual representations. Further alignment is obtained in two ways. One is through network alignment between pretraining and downstream object detection, such that all layers of the detectors can be well-initialized. The other is by accounting for important properties of object detection such as object-level translation invariance and scale invariance. SoCo achieves state-of-the-art transfer performance on COCO detection using a Mask R-CNN detector. Experiments on two-stage and single-stage detectors demonstrate the generality and extensibility of SoCo.
title: Aligning Pretraining for Detection via Object-Level Contrastive Learning