summaries: This paper presents a hierarchical cross contrastive self-supervised learning framework for learning visual representation. This paper proposes to project the representations of an image and its augmented version to multiple latent spaces and also make predictions on each of the latent spaces.  A contrastive loss between the features of different projection levels is minimized to learn the parameters. Experiments on the image classification and detection benchmarks are evaluated. A comparison between important existing methods is also done in the paper.  In this paper, the authors proposed a Hierarchical Cross Contrastive Learning (HCCL) method for Self-supervised learning (SSL) of visual representation. The proposed method include a design of a hierarchical projection network that produces multi-level latent representations. A cross contrastive loss is also introduced to learn invariant visual representations. HCCL is validated on several downstream tasks including classification, segmentation and object detection. The authors propose an extension to the contrastive learning based representation learning approach. The proposed method, call Hierarchical Cross Contrastive Learning(HCCL), leverages the features in different levels and views for more consistent features over the standard CL using only the features of the final layers. The effectiveness is validated in classification, detection, segmentation, and few-shot learning tasks.  This manuscript proposed a new contrastive self-supervised learning approach (HCCL). Compared with exiting work such as BYOL and SimSiam, HCCL introduced (1) multiple hierarchical projectors and predictors; and (2) contrastive loss calculated across different layers of projectors. HCCL is empirically evaluated on several self-supervised learning benchmarks (e.g., iNat18, Place-205, and COCO instance seg, etc.) and achieves noticeable improvement compared with previous states of the art (e.g., SWAV, BYOL, SimSiam, and Barlow Twins). Additionally, ablation studies are provided to verify the significance of hierarchical projectors, cross contrastive loss, and higher learning rates of the predictor.
abstractText: The rapid progress of self-supervised learning (SSL) has greatly reduced the labeling cost in computer vision. The key idea of SSL is to learn invariant visual representations by maximizing the similarity between different views of the same input image. In most SSL methods, the representation invariant is measured by a contrastive loss which compares one of the network outputs after the projection head to its augmented version. Albeit being effective, this approach overlooks the information containing in the hidden layer of the projection head therefore could be sub-optimal. In this work, we propose a novel approach termed Hierarchical Cross Contrastive Learning(HCCL) to further distill the information mismatched by the conventional contrastive loss. The HCCL uses a hierarchical projection head to project the raw representations of the backbone into multiple latent spaces and then compares latent features across different levels and different views. By cross-level contrastive learning, HCCL not only regulates invariant on multiple hidden levels but also crosses different levels, improving the generalization ability of the learned visual representations. As a simple and generic method, HCCL can be applied to different SSL frameworks. We validate the efficacy of HCCL under classification, detection, segmentation, and few-shot learning tasks. Extensive experimental results show that HCCL outperforms most previous methods in various benchmark datasets.
references.authors: Yuki Markus Asano Christian Rupprecht Andrea Vedaldi
references.title: Self-labelling via simultaneous clustering and representation learning
references.venue: arXiv preprint arXiv:1911.05371,
references.authors: Miguel A Bautista Artsiom Sanakoyeu Ekaterina Tikhoncheva Bjorn Ommer
references.title: Cliquecnn: Deep unsupervised exemplar learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Mathilde Caron Piotr Bojanowski Armand Joulin Matthijs Douze
references.title: Deep clustering for unsupervised learning of visual features
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Mathilde Caron Piotr Bojanowski Julien Mairal Armand Joulin
references.title: Unsupervised pre-training of image features on non-curated data
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin
references.title: Unsupervised learning of visual features by contrasting cluster assignments
references.venue: In Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS),
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A simple framework for contrastive learning of visual representations
references.venue: In International conference on machine learning,
references.authors: Xinlei Chen Kaiming He
references.title: Exploring simple siamese representation learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Marco Cuturi
references.title: Sinkhorn distances: Lightspeed computation of optimal transport
references.venue: Advances in neural information processing systems,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: IEEE conference on computer vision and pattern recognition,
references.authors: Carl Doersch Abhinav Gupta Alexei A Efros
references.title: Unsupervised visual representation learning by context prediction
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Jeff Donahue Philipp Krähenbühl Trevor Darrell
references.title: Adversarial feature learning
references.venue: arXiv preprint arXiv:1605.09782,
references.authors: Alexey Dosovitskiy Jost Tobias Springenberg Martin Riedmiller Thomas Brox
references.title: Discriminative unsupervised feature learning with convolutional neural networks
references.venue: Advances in neural information processing systems,
references.authors: Vincent Dumoulin Ishmael Belghazi Ben Poole Olivier Mastropietro Alex Lamb Martin Arjovsky Aaron Courville
references.title: Adversarially learned inference
references.venue: arXiv preprint arXiv:1606.00704,
references.authors: Mark Everingham Luc Van Gool Christopher KI Williams John Winn Andrew Zisserman
references.title: The pascal visual object classes (voc) challenge
references.venue: International journal of computer vision,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: Advances in neural information processing systems,
references.authors: Priya Goyal Piotr Dollár Ross Girshick Pieter Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia Kaiming He
references.title: Accurate, large minibatch sgd: Training imagenet in 1 hour
references.venue: arXiv preprint arXiv:1706.02677,
references.authors: Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre H Richemond Elena Buchatskaya Carl Doersch Bernardo Avila Pires Zhaohan Daniel Guo Mohammad Gheshlaghi Azar
references.title: Bootstrap your own latent: A new approach to self-supervised learning
references.venue: arXiv preprint arXiv:2006.07733,
references.authors: H. Guo K. Zheng X. Fan H. Yu S. Wang
references.title: Visual attention consistency under image transforms for multi-label image classification
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Raia Hadsell Sumit Chopra Yann LeCun
references.title: Dimensionality reduction by learning an invariant mapping
references.venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06),
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Kaiming He Georgia Gkioxari Piotr Dollár Ross Girshick
references.title: Mask r-cnn
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross Girshick
references.title: Momentum contrast for unsupervised visual representation learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Yuenan Hou Zheng Ma Chunxiao Liu Chen Change Loy
references.title: Learning lightweight lane detection cnns by self attention distillation
references.venue: In Proceedings of the IEEE/CVF international conference on computer vision,
references.authors: Zeyi Huang Yang Zou Vijayakumar Bhagavatula Dong Huang
references.title: Comprehensive attention selfdistillation for weakly-supervised object detection
references.venue: arXiv preprint arXiv:2010.12023,
references.authors: Diederik P Kingma Max Welling
references.title: Auto-encoding variational bayes
references.venue: arXiv preprint arXiv:1312.6114,
references.authors: Gustav Larsson Michael Maire Gregory Shakhnarovich
references.title: Learning representations for automatic colorization
references.venue: In European conference on computer vision,
references.authors: Tsung-Yi Lin Michael Maire Serge Belongie James Hays Pietro Perona Deva Ramanan Piotr Dollár C Lawrence Zitnick
references.title: Microsoft coco: Common objects in context
references.venue: In European conference on computer vision,
references.authors: Ilya Loshchilov Frank Hutter
references.title: Sgdr: Stochastic gradient descent with warm restarts
references.venue: arXiv preprint arXiv:1608.03983,
references.authors: Ishan Misra Laurens van der Maaten
references.title: Self-supervised learning of pretext-invariant representations
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Ishan Misra C Lawrence Zitnick Martial Hebert
references.title: Shuffle and learn: unsupervised learning using temporal order verification
references.venue: In European Conference on Computer Vision,
references.authors: Mehdi Noroozi Paolo Favaro
references.title: Unsupervised learning of visual representations by solving jigsaw puzzles
references.venue: In European conference on computer vision,
references.authors: Deepak Pathak Philipp Krahenbuhl Jeff Donahue Trevor Darrell Alexei A Efros
references.title: Context encoders: Feature learning by inpainting
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Pedro O Pinheiro Amjad Almahairi Ryan Y Benmalek Florian Golemo Aaron Courville
references.title: Unsupervised learning of dense visual representations
references.venue: arXiv preprint arXiv:2011.05499,
references.authors: Shaoqing Ren Kaiming He Ross Girshick Jian Sun
references.title: Faster r-cnn: Towards real-time object detection with region proposal networks
references.venue: Advances in neural information processing systems,
references.authors: Byungseok Roh Wuhyun Shin Ildoo Kim Sungwoong Kim
references.title: Spatially consistent representation learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Dawei Sun Anbang Yao Aojun Zhou Hao Zhao
references.title: Deeply-supervised knowledge synergy
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Yonglong Tian Dilip Krishnan Phillip Isola
references.title: Contrastive multiview coding
references.venue: In Computer Vision–ECCV 2020: 16th European Conference,
references.authors: Naftali Tishby Noga Zaslavsky
references.title: Deep learning and the information bottleneck principle
references.venue: In 2015 IEEE Information Theory Workshop (ITW),
references.authors: Naftali Tishby Fernando C Pereira William Bialek
references.title: The information bottleneck method
references.venue: arXiv preprint physics/0004057,
references.authors: Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona Serge Belongie
references.title: The inaturalist species classification and detection dataset
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Pascal Vincent Hugo Larochelle Yoshua Bengio Pierre-Antoine Manzagol
references.title: Extracting and composing robust features with denoising autoencoders
references.venue: In Proceedings of the 25th international conference on Machine learning,
references.authors: Lezi Wang Ziyan Wu Srikrishna Karanam Kuan-Chuan Peng Rajat Vikram Singh Bo Liu Dimitris N Metaxas
references.title: Sharpen focus: Learning with attention separability and consistency
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Xinlong Wang Rufeng Zhang Chunhua Shen Tao Kong Lei Li
references.title: Dense contrastive learning for self-supervised visual pre-training
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Zhirong Wu Yuanjun Xiong Stella X Yu Dahua Lin
references.title: Unsupervised feature learning via nonparametric instance discrimination
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Enze Xie Jian Ding Wenhai Wang Xiaohang Zhan Hang Xu Zhenguo Li Ping Luo
references.title: Detco: Unsupervised contrastive learning for object detection
references.venue: arXiv preprint arXiv:2102.04803,
references.authors: Yang You Igor Gitman Boris Ginsburg
references.title: Scaling sgd batch size to 32k for imagenet training
references.venue: arXiv preprint arXiv:1708.03888,
references.authors: Jure Zbontar Li Jing Ishan Misra Yann LeCun Stéphane
references.title: Deny. Barlow twins: Self-supervised learning via redundancy reduction
references.venue: arXiv preprint arXiv:2103.03230,
references.authors: Xiaohua Zhai Avital Oliver Alexander Kolesnikov Lucas Beyer
references.title: S4l: Self-supervised semisupervised learning
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Richard Zhang Phillip Isola Alexei A Efros
references.title: Colorful image colorization
references.venue: In European conference on computer vision,
references.authors: Bolei Zhou Agata Lapedriza Jianxiong Xiao Antonio Torralba Aude Oliva
references.title: Learning deep features for scene recognition using places database
references.venue: Advances in Neural Information Processing Systems,
references.authors: Chengxu Zhuang Alex Lin Zhai Daniel Yamins
references.title: Local aggregation for unsupervised learning of visual embeddings
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
sections.heading: 1 INTRODUCTION
sections.text: Self-supervised learning (SSL) of visual representation aims to learn representations from image pixels without semantic labels. Recently, contrastive representation learning has gained growing attention by demonstrating state-of-the-art performance in SSL for large-scale image recognition, closing the gap to supervised baselines (Chen et al., 2020; Grill et al., 2020; Chen & He, 2021). Contrastive representation learning is essentially an instance-level discrimination task. It narrows the distance of the representation generated by the same image, as an instance class, while pushing away the representation generated by other images to learn invariant visual representations. In most contrastive learning methods, the representation invariant is measured by a contrastive loss, which compares one of the network outputs after the projection head to its augmented version. Some works attempt to explore advanced training methods to keep the contrastive loss stable in SSL, by clustering, memory bank, momentum update or projector head(Caron et al., 2020; He et al., 2020; Grill et al., 2020). Some other works seek to improve representation performance by increasing transformation views with complicated augmentations. In this paper, we focus on improving contrastive learning by studying the projection head.
  As shown above, the projection head serves as a bridge to squeeze out redundant information in order to retrain invariant information from the backbone for the contrastive loss module. However, only the last output of the projection head is used, while hidden layers are overlooked, which loses many valuable information according to the information bottleneck theory(Tishby et al., 2000; Tishby & Zaslavsky, 2015). Inspired by the successful results of knowledge distillation in supervised representation learning, we propose a cross-level contrastive learning method for SSL(Sun et al., 2019). In this framework, contrastive losses are applied in the intermediate layers of projection heads. In particular, a cross-layer assignment is adapted to create pairs between different views, as the representation generated at the same level of projection heads of different transformed views is likely to retrain similar information, which is not conducive to learning extra knowledge through contrast-
  ing. The consistency assumption over the representation in different views and different layers has been partially explored in many feature regularization approaches(Guo et al., 2020; Hou et al., 2019; Wang et al., 2019; Huang et al., 2020). This phenomenon is investigated in BYOL, which shows that the use of different projection heads for contrastive learning can achieve better performance.
  In this paper, a series of experiments and analyses are proposed to validate our contributions. The experimental results show the effectiveness of our method in improving the robustness of the pretraining model for downstream tasks. Specifically, in the ImageNet(Deng et al., 2009) linear evaluation task, the standard ResNet-50(He et al., 2016) trained by our method achieves 73.5% top-1 accuracy. In the object detection and instance segmentation tasks, our method also outperforms random initialization and existing image-level self-supervision methods on PASCAL VOC(Everingham et al., 2010) and COCO(Lin et al., 2014) datasets. Last but not the least, compared to other SSL methods, HCCL requires fewer training epoch to learn effective representations. For example, it only takes 100 epochs to achieve 69.5% top-1 accuracy in the ImageNet linear evaluation task, while other methods may take 200 epochs or more. Our main contributions can be summarized as follows:
  • In this paper, we propose a Hierarchical Cross Contrastive Learning (HCCL) method to explore information from hidden layers of the projection head. Experiments show that our method can achieve state-of-the-art results in a series of standard benchmarks.
  • Ablation studies are proposed to evaluate the modules of CLIC. Experiments show that cross-level assignments, multi-layer outputs and multi-predictors support the final performance improvements.
  • We evaluate the generalization of HCCL method in both SimSiam and BYOL frameworks. The experiments show that HCCL can significantly improve performance and speed up convergence.
sections.heading: 2 RELATED WORK
sections.text: Self-supervised learning or unsupervised visual representation learning has undergone a relatively long period of development. Early works use generative methods to restore the distribution of input data, usually including auto-encoders (Kingma & Welling, 2013; Vincent et al., 2008) and adversarial learning (Goodfellow et al., 2014; Donahue et al., 2016; Dumoulin et al., 2016). However, these methods are intended to restore the high-level details of the image that are not necessary for downstream tasks. On the other hand, pixel-level reconstruction also costs a lot of calculations.
  Some other methods learn specific representations by handcrafted pretext tasks. These pretext tasks include colorization(Zhang et al., 2016; Larsson et al., 2016), jigsaw puzzle(Noroozi & Favaro, 2016; Misra et al., 2016), relative patch prediction (Doersch et al., 2015), inpainting(Pathak et al., 2016), geometric transformations(Dosovitskiy et al., 2014), etc. However, the design of excuse tasks relies on strong prior knowledge, and the representations learned by these methods usually lack generality in downstream tasks.
  Another type of works is the clustering based self-training (Bautista et al., 2016; Asano et al., 2019; Caron et al., 2018; 2019; 2020). In general, they cluster the representations and learn to predict the cluster assignment. DeepCluster (Caron et al., 2018; 2019) uses k-means to create pseudo labels for samples. It clusters data using previous versions of representation and uses the cluster index of each sample as a classification target for the new representation. SwAV calculates the assignment from one view and predicts it from another (Caron et al., 2020). It produces soft assignment by the Sinkhorn-Knopp algorithm (Cuturi, 2013) and performs online clustering under a balanced partition constraint for each batch. Clustering-based methods require a costly clustering phase to avoid collapsing, and a large memory bank or large batches to provide sufficient samples for clustering.
  Contrastive learning measures the similarity of pairs of samples in the representation space by using contrast loss(Hadsell et al., 2006). It brings the representation of different views of the same sample closer and pushes the representation of views from different samples away. Dosovitskiy et al. (2014) considers each image in a dataset as its own class represented by a feature vector and separates each image. Since this method produces a huge classification header that is difficult to handle, Wu et al. (2018) use a memory bank that stores previously-computed instance representation vectors to replace the classifier. Several later works also follow and extend this approach(Misra & Maaten,
  2020; Tian et al., 2020; Zhuang et al., 2019). MoCo(He et al., 2020) reduces the size of the memory bank by maintaining a dynamic queue that stores negative sample representations. The representation in the queue is generated by a moving average encoder to improve consistency with the query representation. SimCLR(Chen et al., 2020) removes the memory bank and uses the negative samples that exist in the current batch directly for learning. It introduces nonlinear projection heads and complex composition of image transformations to improve the robustness of representations in downstream tasks. BYOL(Grill et al., 2020) further examines the need for negative samples, using a slow-moving average target network to produce stable targets for the online network. The online network bootstraps its own representation by keeping pace with stable targets. By analysing certain conditions for avoiding collapsing solutions, SimSiam(Chen & He, 2021) removed the momentum encoder on the basis of BYOL. BARLOW TWINS(Zbontar et al., 2021) proposes an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two siamese networks fed with different views of a sample and bringing it as close as possible to the identity matrix. It causes the representation of different views of a sample to be similar, while minimizing the redundancy between the components of these representations. The abovementioned self-supervised learning methods all express learning as image-level predictions using global representations, and the gap between image-level pre-training and dense prediction tasks still exists. Some works propose pixel-level self-supervised learning methods to improve the effect of pre-training representations on dense prediction tasks(Wang et al., 2021; Roh et al., 2021; Xie et al., 2021; Pinheiro et al., 2020).
sections.heading: 3 METHOD
sections.text: In this section, we describe the proposed HCCL in detail. First, we briefly introduce the pipeline of the proposed HCCL shown in Figure 1, including the design of a hierarchical projection network that maps representations to different latent spaces. Then we introduce the cross contradictive loss, which allows explicit information interactions between different latent layers. Finally, we present the implementation details in HCCL, such as image transforms, architecture, and optimization.
sections.heading: 3.1 FRAMEWORK
sections.text: HCCL builds on the cross-view prediction framework introduced in recent successful contrastive learning approaches(Chen et al., 2020; Grill et al., 2020; Chen & He, 2021). The cross-view prediction framework is based on a prediction problem that the representation of a transformed view of an image should be predictive of the representation of another transformed view of the same image. HCCL transforms an image X randomly to two distorted views Xa and Xb, by applying transformations sampled from the set T of image transformations. The two distorted views Xa and Xb
  are then fed to non-linear mapping encoder fθ, typically a deep network with trainable parameters θ, producing two raw representations yaand yb respectively. For fθ, HCCL allows various choices of the network architecture without any constraints. yaand yb are representations used for transfer tasks. After extracting the raw representation ya and yb, a hierarchical neural network projection head gnθ that maps them to different latent space, produces n embedding Zna = {za1, za2, . . . ,zan} and Znb = {zb1, zb2, . . . ,zbn}. Further, n prediction MLP head, denoted as h1, h2, . . . , hn, transforms the projection embedding Zna and Znb to 2n vectors as Pna = {pa1,pa2, . . . ,pan} and Pnb = {pb1,pb2, . . . ,pbn}. The following formulas summarizes the above pipeline:
  Zna = gnθ (fθ(Xa)) (1)
  Znb = gnθ (fθ(Xb)) (2) pai = hi(zai) (3)
  pbi = hi(zbi) (4)
  where ya = fθ(Xa), yb = fθ(Xb) and i ∈ {1, 2, . . . , n}. Follow the same way as SimSiam(Chen & He, 2021) and BYOL(Grill et al., 2020), HCCL matches one view’s prediction vector to the other view’s projection embedding finally. The difference is that we have multiple projection embedding and prediction vectors, and cannot directly use the original contrastive loss, so we designed a cross contrastive loss to solve it.
sections.heading: 3.2 CROSS CONTRASTIVE LOSS
sections.text: Since Zn are extracted from different levels of the projection head, the high-level representations are more invariant and the low-level representations are more informative. HCCL adopts an intuitive approach, that is, cross contrastive representations of different levels to trade off invariant and informative. Given the projection embedding z and the prediction vector p, we minimize their mean squared error of l2-normalized vectors:
  D(p, z) = − p · z ||p||2 · ||z||2
  (5)
  where || · ||2 is l2-norm. After that, we define the cross contrastive loss as:
  Lcross(Pn,Zn) = D(p1, zn) + n∑ i=2 D(pi, zi−1) (6)
  Here each level of prediction vector contrast with the previous level of projection representation except the first level one. The first level of the prediction vector contrast with the final level of projection representation. Following BYOL and SimSiam, We symmetrize the loss Lcross in Eq.6 and take a stop-gradient operation to projection embedding Zn. The form of total loss is implemented as: L = Lcross(Pna , stopgrad(Znb )) + Lcross(Pnb , stopgrad(Zna)) (7) Here the encoder only receives gradient from branch Pa and Pbin the two terms. The pseudo-code is provided in Algorithm refalgorithm
sections.heading: 3.3 IMPLEMENTATION DETAILS
sections.text: Image augmentations We use the same image augmentation set and parameters as BYOL(Grill et al., 2020). Firstly, two different views are randomly cropped from each input image and resized to 224 × 224, followed by horizontal flipping, color distortion, converting to grayscale, Gaussian blurring, and polarization. The color distortion consists of a random sequence of brightness, contrast, saturation, hue adjustments. Random cropping and resizing are always applied, others are applied randomly with a certain probability. For gaussian blurring and polarization, the probabilities applied on the two distorted views are different.
  Architecture We use ResNet-50(He et al., 2016) as our base parametric encoder network, the fullyconnected layer after global average pooling is removed and replaced by a hierarchical projection network. Specifically, We adopt a 2-level projector network that projects the output of the global average pooling layer to two latent spaces. Each level projector network has three linear layers and each layer has 2048 output units. The first two linear layers of each level projector are followed
  Algorithm 1 PyTorch-style pseudocode for HCCL
  # f: encoder networks # g: hierarchical projetcion head(with two levels) # h1, h2: two prediction networks # n: batch size
  for x in loader: # load a minibatch x with n samples x_a, x_b = augment(x) # two augmented views y_a, y_b = f(x_a), f(x_b) # encoder output z_a1, z_a2 = g(y_a) # proj outputs of view a z_b1, z_b2 = g(y_b) # proj outputs of view b p_a1, p_a2 = h1(z_a1), h2(z_a2) # pred outputs of view a p_b1, p_b2 = h1(z_b1), h2(z_b2) # pred outputs of view b loss = [L(p_a1, z_b2)/2 + L(p_a2, z_b1) + \\
  L(p_b1, z_a2)/2 + L(p_b2, z_a1)]/2 loss.backward() # back-propagate optimizer.step()
  def L(t, s): s = s.detach() # stop gradient t = normalize(t) s = normalize(s) return -(t * s).sum(dim=1).mean()
  by a batch normalization layer and rectified linear units(ReLU). Each level projector network is followed by a prediction network which has two linear layers. The dimension of the prediction network’s input and output is 2048 and the hidden layer’s dimension is 512. The prediction network has batch normalization and ReLU applied to its hidden layer. Its output layer does not have batch normalization or ReLU.
  Optimization We adapt the same optimization protocol described in SimSiam(Chen et al., 2020). We use SGD optimizer(You et al., 2017) with a momentum parameter of 0.9. We use cosine decay learning rate schedule(Loshchilov & Hutter, 2016) and train for 800 epochs with a batch size of 512, without restarts and warm-up. We set the initial learning rate to 0.025 and multiply the learning rate by the batch size and divide it by 256(Goyal et al., 2017). In addition, we use a global weight decay parameter of 1.5·10-4 while excluding the biases and batch normalization parameters from weights. Following SimCLR(Chen et al., 2020) We use batch normalization synchronized across devices. Training is distributed across 8 V100 GPUs and takes approximately 230 hours for a ResNet-50. For 100, 200, 400, and other less epoch training, we set the initial learning rate to 0.05 and weight decay parameter to 1·10-4. Since SimSiam(Chen et al., 2020) and BYOL(Grill et al., 2020) have explained that the predictor network should adapt to the latest representation, so it is not necessary to force it to converge before the representations. In our experiment, we keep the learning rate of one predictor constant and set the learning rate of the other predictor to 10 times the initial learning rate and decay normally.
sections.heading: 4 EXPERIMENTS
sections.text: In this section, we analyze the performance of HCCL’s representations after self-supervised pretraining on the training set of the ImageNet ILSVRC-2012 dataset(Deng et al., 2009) by transfer learning on different datasets and tasks. Firstly, we evaluate the pre-trained representation on ImageNet in both linear classification on frozen features and semi-supervised learning. Then we assess its transfer capabilities on other vision datasets and tasks, including classification, segmentation, and object detection.
sections.heading: 4.1 LINEAR AND SEMI-SUPERVISED EVALUATIONS ON IMAGENET
sections.text: Linear evaluation on ImageNet We first verify our method by training a linear classifier on ImageNet on fixed representations, which are from ResNet’s global average pooling layer. Following a common protocol, the linear classifier is trained for 90 epochs with a batch size of 1024 and a cosine learning rate schedule. We set the initial learning rate to 0.1 and multiply the learning rate by the batch size and divide it by 256. We minimize the cross-entropy loss with the LARS optimizer with a momentum of 0.9 and weight decay of 0. At training time, the input image is randomly cropped, resized to 224×224, and randomly flipped horizontally. At test time, the image is resized to 256×256 and center-cropped to a size of 224×224.
  We compare with the state-of-the-art frameworks in Table 1 on ImageNet linear evaluation. All competitors are based on original papers or reproduction trained in SimSiam(Chen & He, 2021). All competitors are based on a standard ResNet50, with two 224×224 views used during pre-training. Table 1 reports the top-1 accuracies obtained on the ImageNet validation set and the main properties of the methods. HCCL achieves competitive results with a top-1 accuracy of 73.5%, which is comparable to the state-of-the-art methods. HCCL is trained with small batch size and used neither negative samples nor a momentum encoder. It achieved the best results among all methods under 100-epoch and 200-epoch pre-training. HCCL only 0.8% below the performance of BYOL(Grill et al., 2020). BYOL uses a momentum encoder, which means a greater training cost, because it requires 4 times forward in one iteration, while HCCL only requires 2 times. In addition, HCCL can also be flexibly applied to the SSL method with momentum encoder. We evaluated HCCL with momentum encoder, it achieved better performance than BYOL.(See Appendix A.2)
  Semi-supervised learning on ImageNet We simply fine-tune the whole ResNet-50 pre-trained with our method on a small subset of ImageNet’s train set and evaluate the performance. We sample
  1% and 10% of the labeled ImageNet training datasets in a class-balanced way by following SimCLR(Chen et al., 2020). The image augmentations are the same as in the linear evaluation setting. We train for 20 epochs with a batch size of 256 and a momentum of 0.9. We minimize the crossentropy loss with the SGD optimizer with Nesterov momentum and do not use any regularization methods. We use a learning rate of 0.02 for the conv-net weights and 300 for the final linear layer, and we decay the learning rates with a cosine learning rate schedule. Table 2 shows the comparisons of our semi-supervised results against recent methods. We report both top-1 and top-5 accuracies on the ImageNet validation set. HCCL is either on par or slightly outperforms previous competing semi-supervised and semi-supervised approaches.
sections.heading: 4.2 TRANSFER TO OTHER DATASETS AND TASKS
sections.text: Image classification with fixed features We follow the exact settings from PIRL(Misra & Maaten, 2020) for training and evaluating linear classifiers on the iNaturalist2018(Van Horn et al., 2018), Places205(Zhou et al., 2014) and VOC07(Everingham et al., 2010) datasets. For VOC07, We train a linear SVM on the global average pooled final representations. For iNaturalist2018 and Places-205, we train a linear classifier with SGD using a learning rate of 0.01 reduced by a factor of 10 at two equally spaced intervals, a batch size of 256, a weight decay of 0.0001, and SGD momentum of 0.9. We train the linear models for 30 epochs on Places-205 and 84 epochs on iNat18. We report the top-1 accuracy computed using the 224 × 224 center crop on the validation set. The results in Table 3 show that HCCL almost outperforms all prior works on three datasets.
  Object Detection and Instance Segmentation We evaluate our representations by transferring them to localization-based tasks, like object detection and instance segmentation. Following the setup in MOCO(He et al., 2020), we fine-tune all layers of the pre-trained models end-to-end in the VOC07+12(Everingham et al., 2010) and COCO(Lin et al., 2014) datasets. We use Faster RCNN(Ren et al., 2015) with an R50-C4 backbone on VOC and use Mask R-CNN(He et al., 2017) with an R50-C4 backbone on COCO.
  We use the detection models as implemented in Detectron2 library(Wu et al., 2019) for training and closely follow the finetune and evaluation settings from the public codebase of MoCo. For Pascal VOC, We use the VOC07+12 trainval set for training, and report results on the VOC07 test set averaged over 5 independent runs. We train a Faster R-CNN C4 backbone for 24K iterations using a batch size of 16 across 8 GPUs using SyncBatchNorm. We set the initial learning rate to 0.1 and reduce it by a factor of 10 after 18K and 22K iterations. We use linear warmup(Goyal et al., 2017) with a slope of 0.333 for 1000 iterations. For COCO, We use the COCO 2017 train split for training and report results on the val split averaged over 5 independent runs. We set the initial learning rate to 0.03 and keep the other parameters the same as in the 1× schedule in MoCo’s codebase.
  In Table4, we show that HCCL outperforms among these leading methods on both Pascal VOC and COCO datasets. It means that HCCL’s representations are transferable beyond the classification task.
sections.heading: 5 ABLATIONS
sections.text: We present extensive ablation experiments on HCCL to give an intuition of its behavior and performance and show how each component contributes. We train HCCL for 100 epochs among all ablation experiments, which performs consistent results compared to our baseline training of long epochs. For reproducibility, all the experiment results are averaged over 3 independent trials. For all the experiments in this section, we set the initial learning rate to 0.05 with batch size 512, the weight decay to 10-4 as in SimSiam(Chen et al., 2020). We report the top-1 accuracy of training linear classifiers on the ImageNet under the linear evaluation protocol the same as in Section 4.
sections.heading: 5.1 INFLUENCE OF HIERARCHICAL PROJECTION HEAD AND CROSS CONTRASTIVE LOSS
sections.text: In this subsection, we alter our hierarchical projection head and cross contrastive loss in several ways to test the necessity of each term of them. Figure 2 recapitulates the different structure and learning ways on projection head and Table 5 shows their results on a linear evaluation benchmark of Imagenet.
  Deep projection with single level We first confirm that the gain of HCCL is not brought about by a deep projection head with a single level. In Figure 2(a), we use a 6 layer projection head the same as in section 4, only single-level representation (the output of the final linear layer) are used to construct the contrastive loss. Compared to HCCL, the performance drops by 0.8%, which means that the lack of hierarchical representation and cross contrastive loss will cause performance degradation.
  Hierarchical projection head without cross contrastive loss In Figure 2(b), we use a 6 layer projection head the same as in section 4, and multi-level representation (the output of the final linear layer and the 3rd linear layer) are used to construct the contrastive loss. Instead of cross contrastive between different levels, we contrast representations of the same level with each other. Compared with HCCL the performance reduced by 0.8%, which means that hierarchical representations must be combined with a cross contrastive loss to improve performance.
  Hierarchical projection head with single cross contrastive loss Figure 2(c) and Figure 2(b) use the same hierarchical projection structure, the difference is that in Figure 2(c) we use the highlevel feature of view1 and the low-level feature of view2 for cross contrastive. Compared with the
  standard HCCL structure, the cross contrastive between the low-level feature of view1 and the highlevel of view2 is removed. It brings 1.0% drops compared with HCCL. Obviously that only a oneway contrastive between high-level and low-level representations cannot improve the performance.
  Hierarchical projection head with single predictor In Figure 2(d), we share a single predictor among all level projections, which brings 0.7% drops. The output representations of different levels of projection heads are distributed in different spaces, and the use of a shared single predictor will destroy this distribution and lead to worse performance.
  Deep projection with multi predictor We confirm that the gain of HCCL is not brought about by multi predictor. In Figure 2(e), we use a 6 layer projection head the same as in section 4. The output of the projection head is fed to two different predictors, and then the contrastive loss is constructed. Compared to HCCL, the performance drops by 1.1%.
sections.heading: 5.2 LEVELS AND DEPTH OF PROJECTOR NETWORK
sections.text: Table 6 shows the influence of projector architecture on HCCL. We examine the effect of different levels and depths for the projector network. Using the default projector network of level 2 and depth 3 yields the best performance. In addition, we find that our model performs worse when the projector network has more layers, with a saturation of the performance for 3 layers and 2 levels.
sections.heading: 5.3 PREDICTOR LEARNING RATE
sections.text: In this subsection, we examine the effect of the combination of learning rates used by different predictors in HCCL. BYOL(Grill et al., 2020) and SimSiam(Chen & He, 2021) explains that predictors should adapt to the latest representations and keep near-optimal, so use a sufficiently large learning rate of predictor provides a reasonably good performance. We use the default HCCL structure of level 2 and depth 3, with 2 different predictors. We multiply the learning rate of the predictors by a constant λ compared to the learning rate used for the rest of the network; all other hyper-parameters are unchanged. Table 7 provides the performance of the two predictors using constant learning rate or different λ times learning rate. Results show that the predictor learning rate needs to be much higher than the projector learning rate.
sections.heading: 6 CONCLUSION
sections.text: We propose HCCL, a simple yet effective method strategy that applies to a class of self-supervised learning. The main idea of HCCL is to explore multiple outputs from hidden layers instead of the output of the last one, and to compare latent features on different levels and views. We show that HCCL regulates not only invariant on multiple hidden levels, but also across different levels, improving the generalization ability of the learned visual representations. We also show that HCCL can work in various SSL frameworks like BYOL and Simsiam. The results of the experiments suggest that HCCL is particularly effective when computing resources are limited, which is widespread in practice.
sections.heading: A APPENDIX
sections.text: A.1 THE PROOF OF HCCL’S CONVERGENCE
  We consider a deep linear system where x ∈ Rd is the input image, W ∈ Rm×d extract ddimensional latent features from x. The augmented version of x is x̂. The fc layer wfc generates the final label y = wfc>Wx. The ground-truth is denoted as y∗ = w∗fc
  >W ∗x. The matrix Ŵ1 ∈ Rm×d denotes the stopping-gradient copy of W1 ∈ Rm×d, Ŵ1 = StopGrad(W1). Similary, W2, Ŵ2,P1,P2, P̂1, P̂2 ∈ Rm×m. The learning rates for {W1,W2} and {P1P2} are {ηw, ηp} respectively. With out loss of generality, we assume the features of x can be divided into two parts x = [x1,x2] where x1 corresponds to the augmentation invariant part and x2 corresponds to the augmentation parts. Similarily, x̂ = [x1; x̂2]. We futher assume that the invariant part and augmentation part are independent and the augmentation is i.i.d. generated, that is, E{xx>} = Id×d, E{x1x1>} = Im×m, E{x2x2>} = I(d−m)×(d−m), E{x1x2>} = 0, E{x1x̂2>} = 0, E{x2x̂2>} = 0. Aligned with x, matrix W can be divided into two parts too, W = [U ;V ]. Ideally, ‖U∗‖ 0 and ‖V ∗‖ = 0 gives the optimal solution. The measure the quality of self-supervised learning, we define ρ = ‖V ‖/‖U‖ as the recovering error of invariant feature subspace. ρ = 0 means that the SSL algorithm finds the perfect Wwithout lossing any userful information.
  In our method, the loss function is L =Ex,x̂ { 1
  2 ‖P1W1x− Ŵ2Ŵ1x̂‖2 +
  1 2 ‖P2W2W1x− Ŵ1x̂‖2 } The gradients are
  ∇P1L =Ex,x̂ { (P1W1x− Ŵ2Ŵ1x̂)x>W1> } =P1W1W1 > − Ŵ2Ŵ1diag(I;0)W1>
  =P1(UU > + V V >)−W2UU>
  ∇P2L =Ex,x̂ { (P2W2W1x− Ŵ1x̂)x>W1>W2> } =P2W2W1W1 >W2 > − Ŵ1diag(I;0)W1>W2>
  =P2W2(UU > + V V >)W2 > −UU>W2>
  ∇W1L =Ex,x̂{P1>(P1W1x− Ŵ2Ŵ1x̂)x>+ +W2 >P2 >(P2W2W1x− Ŵ1x̂)x>}
  =P1 >P1W1 − P1>Ŵ2Ŵ1diag(I;0)+
  +W2 >P2 >P2W2W1 −W2>P2>Ŵ1diag(I;0) =P1 >P1W1 − P1>W2[U ;0] +W2>P2>P2W2W1 −W2>P2>[U ;0]
  ∇W2L =Ex,x̂{P2>(P2W2W1x− Ŵ1x̂)x>W1>} =P2 >P2W2W1W1 > − P2>Ŵ1diag(I;0)W1>
  =P2 >P2W2(UU > + V V >)− P2>UU>
  From ∇W1L , we get
  ∇UL =P1>P1U − P1>W2U +W2>P2>P2W2U −W2>P2>U
  ∇V L =P1>P1V +W2>P2>P2W2V
  The SGD iterations are then
  W (t+1) 1 −W (t) 1 =− ηw∇W1L
  W (t+1) 2 −W (t) 2 =− ηw∇W2L
  P (t+1) 1 − P (t) 1 =− ηp∇P1L
  P (t+1) 2 − P (t) 2 =− ηp∇P2L
  Clearly, the above SGD iterations correspond to highly non-linear differential equation systems in high dimensional space. If we do not make further simplifications, there is little we can do analytically. To get a deeper insight , we consider a special case where all matrices are diagonal ones. This simplification allows us to discuss the dynamics analytically. To ensure that the discussion under such simplification can generalize to the original problem, we verify our main conclusion via numerical simulation.
  To continuous our discussion, we assume that d = 2m, U = diag(u(1), u(2), · · · , u(m)), W2 = diag(w
  (1) 2 , w (2) 2 , · · · , w (m) 2 ), P1 = diag(p (1) 1 , p (2) 1 , · · · , p (m) 1 ), P2 = diag(p (1) 2 , p (2) 2 , · · · , p (m) 2 ).
  To avoid notation clutter, the superscript is omitted in the following discussion when it is clear from the context.
  The differential equations for the above SGD iterations are
  u̇ =ηw(p1w2 − p21 + p2w2 − p22w22)u v̇ =− ηw(p21 + p22w22)v ẇ2 =− ηw(u2 + v2)p22w2 + ηwp2u2
  ṗ1 =− ηp(u2 + v2)p1 + ηpu2w2 ṗ2 =− ηp(u2 + v2)w22p2 + ηpu2w2
  Again, these equations are high-order ones. We use numerical simulation to plot their trajectories. We initialized all variables in {±ξ} where ξ = 0.1 is a small number. The SGD learning rate ηw = 0.1, ηp = 1.0. Since there are 5 variables (including {u, v}), we try all the possible initialization combinations and obtain 32 trajectories in Figure 3. The x-axis is u(t) and the y-axis is v(t) The marker X indicates the termination point of the numerical simulation. It is easy to see that all 32 trajectories converge to limt→∞ v(t) = 0 and limt→∞ u(t) = ±0.5 6= 0. This means that the differential equations indeed converge to the ideal solution where the coefficient of the augmentation subspace converges to zero and the coefficient of the invariant subspace converges to a non-zero fixed point.
  A.2 HCCL WITH MOMENTUM ENCODER
  For a fair comparison with BYOL, we also evaluated HCCL with momentum encoder. In this experiment, HCCL has the same architecture as Figure 1, but the parameters of the two branches are no longer shared. As in BYOL, the online network(the branch with predictor) is defined by a set of weights θ, and is comprised of three stages: an encoder fθ, a hierarchical projection head gnθ and n predictor h1, h2, ..., hn. The target network(the branch without predictor) uses a different set of
  weights ξ, and is comprised of two stages: an encoder fξ, a hierarchical projection head gξ. For encoder and projection head, the target parameters ξ are an exponential moving average of the online parameters θ. Given a target decay rate τ ∈ [0, 1], after each training step we perform the following update: ξ = τξ + (1− τ)θ We set τ = 0.996, other training settings are the same as in section 4. Limited by training resources, we give the results under 100, 200, 400, 800 epoch in Table 8. Compared with BYOL, HCCL with momentum encoder is significantly improved.
  A.3 MEMORY FOOTPRINT AND TRAINING SPEED.
  We compared the memory footprint and speed of SimSiam, BYOL, and HCCL during training. All experiments are trained on 8 x V100 GPU with batchsize of 512 and mixed precision. Table 9 shows that HCCL hardly increases memory footprint and training time compared to SimSiam and BYOL.
  A.4 DIFFERENT WAY OF CROSS-CORRELATION.
  We further study the performance of establishing cross-correlation relationships in different ways. When the level of the hierarchical projection head increases, the cross-correlation combination between different levels will become more complicated. In this subsection, we explored the crosscorrelation combination based on the 3 levels hierarchical projection heads. Table 10 shows that cross-correlation combinations between different levels can help improve performance. With more levels of hierarchical projection architecture, the cross-correlation method will become very complicated and cannot be verified one by one through experiments. We believe that the more cross-level cross-correlation relationships are established, the better the performance will be. We will explore this in future work.
  A.5 ANALYZE OF HIERARCHICAL FEATURE.
  In this subsection, we analyze the features extracted by the hierarchical projection head to further explore the mystery behind HCCL.
  A.5.1 Similarity between features of different levels of hierarchical projection head.
  Using SimSiam and HCCL, we extracted the features of 50000 images on the Imagenet Val dataset. Each image is randomly augmented to obtain two views. For SimSiam, each image can extract
  two features, and we calculate the similarity (cosine distance) of the two features. For HCCL, we use a 2-level hierarchical projection head, so each image can extract four features, including two high-level projectors features and two low-level projector features. We calculate the similarity of high-level features, the similarity of low-level features, and the similarity between high-level and low-level features. Figure 5 shows the cosine distance distribution of all samples in the data set. We observed that the features extracted in the same layer of the projector are very similar in both HCCL and SimSiam, and most of their cosine distances are less than 0.2. For HCCL, there are large differences between low-level and high-level projector features. This is consistent with our view that different layers of the projector extract different information. Intuitively, contrastive learning between too similar features is not a good choice.
  A.5.2 Feature similarity within and between classes.
  We extracted the hierarchical projector features of 50000 images on the Imagenet Val dataset. Each sample extracts low-level and high-level features respectively. For each class, we calculate the average cosine distance between all samples in the class, and the results are shown in Figure 6(a). Then we calculate the average cosine distance between the samples of this class and the samples of other classes, the results are shown in Figure 6(b). We can observe that the average cosine distance within classes is smaller than that between classes, whether it is low-level or high-level projector features. The difference is that the average cosine distance of high-level projector features is greater than that of low-level projector features, both within and between classes. We believe that the high-level projector makes semantic disturbance to the low-level features, making the distribution of features more divergent. HCCL’s hierarchical projection head introduces diversity to the features, which is equivalent to different ’data augmentation’ at the feature level. Finally, We use cross contrast loss to learn invariance in these features that experience different disturbances.