summaries: This work addresses some fundamental questions on Vision Transformers including 1. how are Vision Transformers solving vision tasks? 2. Do they act like convolutions, learning the same inductive biases from scratch? 3. And what is the role of scale in learning these representations? To answer these questions, they investigate the internal representation structure of ViTs and CNN by using representational similarity techniques like CKA (Centered Kernel Alignment).  The paper studies the representation of features learned by convnet and transformers. It evaluates the similarity of representations within and between architectures using Centered Kernel Alignment. It studies models trained on datasets of different sizes. Neural network architectures based on self-attention instead of convolutions (e.g., the Vision Transformer, ViT), have recently be shown to reach similar or better performances on ImageNet classification than CNNs. In this work, the authors investigate whether the "internal representation structure" learned by Vision Transformers is also similar to CNNs. Using Centered Kernel Alignment as main metric to compare internal representations, the authors perform several analyses comparing aspects of three Vision Transformer variants (ViT-B/32, ViT-B/16, ViT-L/16) and two ResNet variants (ResNet50x1, ResNet152x2). The analyses performed look at similarity of representations across different depths, local vs. global information aggregation, importance of skip connections, spatial localization of information and effects of the amount of training data. The analyses reveal several differences between the ViTs and ResNets, indicating that despite similar classification performance, the image processing strategies are quite different for the two model families. The paper presents a comparison study between internal representations of recently Vision Transformers and ResNet networks for image classification, using various established tools: Centered kernel alignment, attention distance analysis, effective receptive fields, representation propagation analysis, spatial localization analysis and linear probes. The also train some new models for additional analysis. The findings show significant difference between the models. In this work, the authors explore how representations learned with vision transformers (ViT) compare with those of convolutional neural networks (resnet). Making use of centered kernel alignment (CKA), they compare the representations learned at different layers of the same mode, between models, and at different spatial positions. They find that ViT tends to learn similar representations  across all layers while resnets tend to show a clear distinction between lower-level and higher-level representations. They also provide experiments showing that residual connections play a crucial role in transformers, i.e. removing a residual connection causes a big change between preceding and following representations. Another interesting phenomenon in ViT is that residual connections tend to preserve information of the classification token during the first half of the model while it tends to change during the second half of the model. In addition, they compare how ViT and ResNets treat spatial information. They find that both process local information in the first layers but ViT uses global information earlier than ResNets. In fact, they show that while local processing is "hardcoded" in convolutions, ViT requires larger training sets to learn such inductive bias. Finally, they show that MLP Mixer is closer to ViT than to ResNet.
abstractText: Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.
authors.name: Maithra Raghu
authors.name: Thomas Unterthiner
authors.name: Simon Kornblith
references.authors: G. Alain Y. Bengio
references.title: Understanding intermediate layers using linear classifier probes
references.venue: arXiv preprint arXiv:1610.01644,
references.authors: I. Bello B. Zoph A. Vaswani J. Shlens Q.V. Le
references.title: Attention augmented convolutional networks
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: S. Bhojanapalli A. Chakrabarti D. Glasner D. Li T. Unterthiner A. Veit
references.title: Understanding robustness of transformers for image classification
references.venue: arXiv preprint arXiv:2103.14586,
references.authors: N. Carion F. Massa G. Synnaeve N. Usunier A. Kirillov S. Zagoruyko
references.title: End-to-end object detection with transformers
references.venue: In European Conference on Computer Vision,
references.authors: M. Caron H. Touvron I. Misra H. Jégou J. Mairal P. Bojanowski A. Joulin
references.title: Emerging properties in self-supervised vision transformers
references.venue: arXiv preprint arXiv:2104.14294,
references.authors: M. Chen A. Radford R. Child J. Wu H. Jun D. Luan I. Sutskever
references.title: Generative pretraining from pixels
references.venue: In International Conference on Machine Learning,
references.authors: X. Chen S. Xie K. He
references.title: An empirical study of training self-supervised vision transformers
references.venue: arXiv preprint arXiv:2104.02057,
references.authors: A. Conneau G. Kruszewski G. Lample L. Barrault M. Baroni
references.title: What you can cram into a single vector: Probing sentence embeddings for linguistic properties
references.venue: In ACL,
references.authors: J.-B. Cordonnier A. Loukas M. Jaggi
references.title: On the relationship between self-attention and convolutional layers
references.venue: arXiv preprint arXiv:1911.03584,
references.authors: C. Cortes M. Mohri A. Rostamizadeh
references.title: Algorithms for learning kernels based on centered alignment
references.venue: The Journal of Machine Learning Research,
references.authors: S. d’Ascoli H. Touvron M. Leavitt A. Morcos G. Biroli L. Sagun
references.title: Convit: Improving vision transformers with soft convolutional inductive biases
references.venue: arXiv preprint arXiv:2103.10697,
references.authors: J. Deng W. Dong R. Socher L.-J. Li K. Li L. Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: IEEE conference on computer vision and pattern recognition,
references.authors: J. Devlin M.-W. Chang K. Lee K. Toutanova. Bert
references.title: Pre-training of deep bidirectional transformers for language understanding
references.venue: arXiv preprint arXiv:1810.04805,
references.authors: A. Dosovitskiy L. Beyer A. Kolesnikov D. Weissenborn X. Zhai T. Unterthiner M. Dehghani M. Minderer G. Heigold S. Gelly
references.title: An image is worth 16x16 words: Transformers for image recognition at scale
references.venue: arXiv preprint arXiv:2010.11929,
references.authors: A. Gretton K. Fukumizu C.H. Teo L. Song B. Schölkopf A.J. Smola
references.title: A kernel statistical test of independence
references.venue: In Nips,
references.authors: A. Kolesnikov L. Beyer X. Zhai J. Puigcerver J. Yung S. Gelly N. Houlsby
references.title: Big transfer (bit): General visual representation learning
references.venue: arXiv preprint arXiv:1912.11370,
references.authors: S. Kornblith M. Norouzi H. Lee G. Hinton
references.title: Similarity of neural network representations revisited
references.venue: In ICML,
references.authors: S. Kornblith H. Lee T. Chen M. Norouzi
references.title: What’s in a loss function for image classification
references.venue: arXiv preprint arXiv:2010.16402,
references.authors: N. Kriegeskorte M. Mur P.A. Bandettini
references.title: Representational similarity analysis-connecting the branches of systems neuroscience
references.venue: Frontiers in systems neuroscience,
references.authors: A. Krizhevsky I. Sutskever G.E. Hinton
references.title: Imagenet classification with deep convolutional neural networks
references.venue: Advances in neural information processing systems,
references.authors: S.R. Kudugunta A. Bapna I. Caswell N. Arivazhagan O. Firat
references.title: Investigating multilingual nmt representations at scale
references.authors: G.W. Lindsay
references.title: Convolutional neural networks as a model of the visual system: past, present, and future
references.venue: Journal of cognitive neuroscience,
references.authors: W. Luo Y. Li R. Urtasun R. Zemel
references.title: Understanding the effective receptive field in deep convolutional neural networks
references.venue: arXiv preprint arXiv:1701.04128,
references.authors: N. Maheswaranathan A.H. Williams M.D. Golub S. Ganguli D. Sussillo
references.title: Universality and individuality in neural dynamics across large populations of recurrent networks
references.venue: Advances in neural information processing systems,
references.authors: A. Merchant E. Rahimtoroghi E. Pavlick I. Tenney
references.title: What happens to bert embeddings during fine-tuning
references.venue: arXiv preprint arXiv:2004.14448,
references.authors: A.S. Morcos M. Raghu S. Bengio
references.title: Insights on representational similarity in neural networks with canonical correlation
references.venue: arXiv preprint arXiv:1806.05759,
references.authors: B. Mustafa A. Loh J. Freyberg P. MacWilliams M. Wilson S.M. McKinney M. Sieniek J. Winkens Y. Liu P. Bui
references.title: Supervised transfer learning at scale for medical imaging
references.venue: arXiv preprint arXiv:2101.05913,
references.authors: M. Naseer K. Ranasinghe S. Khan M. Hayat F.S. Khan M.-H. Yang
references.title: Intriguing properties of vision transformers, 2021
references.authors: T. Nguyen M. Raghu S. Kornblith
references.title: Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth
references.authors: N. Parmar A. Vaswani J. Uszkoreit L. Kaiser N. Shazeer A. Ku D. Tran
references.title: Image transformer
references.venue: In International Conference on Machine Learning,
references.authors: S. Paul P.-Y. Chen
references.title: Vision transformers are robust learners
references.venue: arXiv preprint arXiv:2105.07581,
references.authors: M.E. Peters M. Neumann L. Zettlemoyer W.-t. Yih
references.title: Dissecting contextual word embeddings: Architecture and representation
references.venue: In EMNLP,
references.authors: A. Raghu M. Raghu S. Bengio O. Vinyals
references.title: Rapid learning or feature reuse? towards understanding the effectiveness of maml
references.authors: M. Raghu J. Gilmer J. Yosinski J. Sohl-Dickstein
references.title: Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability
references.venue: arXiv preprint arXiv:1706.05806,
references.authors: M. Raghu C. Zhang J. Kleinberg S. Bengio
references.title: Transfusion: Understanding transfer learning for medical imaging
references.venue: arXiv preprint arXiv:1902.07208,
references.authors: P. Ramachandran N. Parmar A. Vaswani I. Bello A. Levskaya J. Shlens
references.title: Stand-alone self-attention in vision models
references.authors: O. Russakovsky J. Deng H. Su J. Krause S. Satheesh S. Ma Z. Huang A. Karpathy A. Khosla M. Bernstein
references.title: Imagenet large scale visual recognition challenge
references.venue: International journal of computer vision,
references.authors: J. Shi E. Shea-Brown M. Buice
references.title: Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex
references.venue: Advances in Neural Information Processing Systems,
references.authors: L. Song A. Smola A. Gretton J. Bedo K. Borgwardt
references.title: Feature selection via dependence maximization
references.venue: Journal of Machine Learning Research,
references.authors: C. Sun A. Shrivastava S. Singh A. Gupta
references.title: Revisiting unreasonable effectiveness of data in deep learning era
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Y. Tay M. Dehghani J. Gupta D. Bahri V. Aribandi Z. Qin D. Metzler
references.title: Are pre-trained convolutions better than pre-trained transformers
references.venue: arXiv preprint arXiv:2105.03322,
references.authors: I. Tolstikhin N. Houlsby A. Kolesnikov L. Beyer X. Zhai T. Unterthiner J. Yung D. Keysers J. Uszkoreit M. Lucic
references.title: Mlp-mixer: An all-mlp architecture for vision
references.venue: arXiv preprint arXiv:2105.01601,
references.authors: H. Touvron P. Bojanowski M. Caron M. Cord A. El-Nouby E. Grave A. Joulin G. Synnaeve J. Verbeek H. Jégou
references.title: Resmlp: Feedforward networks for image classification with dataefficient training
references.venue: arXiv preprint arXiv:2105.03404,
references.authors: A. Vaswani N. Shazeer N. Parmar J. Uszkoreit L. Jones A.N. Gomez L. Kaiser I. Polosukhin
references.title: Attention is all you need
references.venue: arXiv preprint arXiv:1706.03762,
references.authors: E. Voita R. Sennrich I. Titov
references.title: The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives
references.authors: B. Wu C. Xu X. Dai A. Wan P. Zhang Z. Yan M. Tomizuka J. Gonzalez K. Keutzer P. Vajda
references.title: Visual transformers: Token-based image representation and processing for computer vision
references.venue: arXiv preprint arXiv:2006.03677,
references.authors: J.M. Wu Y. Belinkov H. Sajjad N. Durrani F. Dalvi J. Glass
references.title: Similarity analysis of contextual word representation models
references.venue: arXiv preprint arXiv:2005.01172,
references.authors: S. Wu A. Conneau H. Li L. Zettlemoyer V. Stoyanov
references.title: Emerging cross-lingual structure in pretrained language models
references.authors: L. Yuan Y. Chen T. Wang W. Yu Y. Shi Z. Jiang F.E. Tay J. Feng S. Yan
references.title: Tokens-to-token vit: Training vision transformers from scratch on imagenet
sections.heading: 1 Introduction
sections.text: Over the past several years, the successes of deep learning on visual tasks has critically relied on convolutional neural networks [20, 16]. This is largely due to the powerful inductive bias of spatial equivariance encoded by convolutional layers, which have been key to learning general purpose visual representations for easy transfer and strong performance. Remarkably however, recent work has demonstrated that Transformer neural networks are capable of equal or superior performance on image classification tasks at large scale [14]. These Vision Transformers (ViT) operate almost identically to Transformers used in language [13], using self-attention, rather than convolution, to aggregate information across locations. This is in contrast with a large body of prior work, which has focused on more explicitly incorporating image-specific inductive biases [30, 9, 4]
  This breakthrough highlights a fundamental question: how are Vision Transformers solving these image based tasks? Do they act like convolutions, learning the same inductive biases from scratch? Or are they developing novel task representations? What is the role of scale in learning these representations? And are there ramifications for downstream tasks? In this paper, we study these questions, uncovering key representational differences between ViTs and CNNs, the ways in which these difference arise, and effects on classification and transfer learning. Specifically, our contributions are:
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  • We investigate the internal representation structure of ViTs and CNNs, finding striking differences between the two models, such as ViT having more uniform representations, with greater similarity between lower and higher layers.
  • Analyzing how local/global spatial information is utilised, we find ViT incorporates more global information than ResNet at lower layers, leading to quantitatively different features.
  • Nevertheless, we find that incorporating local information at lower layers remains vital, with large-scale pre-training data helping early attention layers learn to do this
  • We study the uniform internal structure of ViT, finding that skip connections in ViT are even more influential than in ResNets, having strong effects on performance and representation similarity.
  • Motivated by potential future uses in object detection, we examine how well input spatial information is preserved, finding connections between spatial localization and methods of classification.
  • We study the effects of dataset scale on transfer learning, with a linear probes study revealing its importance for high quality intermediate representations.
sections.heading: 2 Related Work
sections.text: Developing non-convolutional neural networks to tackle computer vision tasks, particularly Transformer neural networks [44] has been an active area of research. Prior works have looked at local multiheaded self-attention, drawing from the structure of convolutional receptive fields [30, 36], directly combining CNNs with self-attention [4, 2, 46] or applying Transformers to smaller-size images [6, 9]. In comparison to these, the Vision Transformer [14] performs even less modification to the Transformer architecture, making it especially interesting to compare to CNNs. Since its development, there has also been very recent work analyzing aspects of ViT, particularly robustness [3, 31, 28] and effects of self-supervision [5, 7]. Other recent related work has looked at designing hybrid ViT-CNN models [49, 11], drawing on structural differences between the models. Comparison between Transformers and CNNs are also recently studied in the text domain [41].
  Our work focuses on the representational structure of ViTs. To study ViT representations, we draw on techniques from neural network representation similarity, which allow the quantitative comparisons of representations within and across neural networks [17, 34, 26, 19]. These techniques have been very successful in providing insights on properties of different vision architectures [29, 22, 18], representation structure in language models [48, 25, 47, 21], dynamics of training methods [33, 24] and domain specific model behavior [27, 35, 38]. We also apply linear probes in our study, which has been shown to be useful to analyze the learned representations in both vision [1] and text [8, 32, 45] models.
sections.heading: 3 Background and Experimental Setup
sections.text: Our goal is to understand whether there are differences in the way ViTs represent and solve image tasks compared to CNNs. Based on the results of Dosovitskiy et al. [14], we take a representative set of CNN and ViT models — ResNet50x1, ResNet152x2, ViT-B/32, ViT-B/16, ViT-L/16 and ViT-H/14. Unless otherwise specified, models are trained on the JFT-300M dataset [40], although we also investigate models trained on the ImageNet ILSVRC 2012 dataset [12, 37] and standard transfer learning benchmarks [50, 14]. We use a variety of analysis methods to study the layer representations of these models, gaining many insights into how these models function. We provide further details of the experimental setting in Appendix A.
  Representation Similarity and CKA (Centered Kernel Alignment): Analyzing (hidden) layer representations of neural networks is challenging because their features are distributed across a large number of neurons. This distributed aspect also makes it difficult to meaningfully compare representations across neural networks. Centered kernel alignment (CKA) [17, 10] addresses these challenges, enabling quantitative comparisons of representations within and across networks. Specifically, CKA takes as input X ∈ Rm×p1 and Y ∈ Rm×p2 which are representations (activation matrices), of two layers, with p1 and p2 neurons respectively, evaluated on the same m examples. Letting K = XXT and L = Y Y T denote the Gram matrices for the two layers (which measures the similarity of a pair of datapoints according to layer representations) CKA computes:
  CKA(K,L) = HSIC(K,L)√
  HSIC(K,K)HSIC(L,L) , (1)
  0 20 40 60 80 100 120 140 Layers ViT-L/16
  0
  20
  40
  60
  80
  100
  120
  140
  La ye
  rs V
  iT -L
  /1 6
  ViT-L/16
  0 25 50 75 100 125 150 175 Layers ViT-H/14
  0
  25
  50
  75
  100
  125
  150
  175
  La ye
  rs V
  iT -H
  /1 4
  ViT-H/14
  0 20 40 60 80 100 120 Layers R50
  0
  20
  40
  60
  80
  100
  120
  La ye
  rs R
  50
  R50
  0 50 100 150 200 250 300 350 Layers R152
  0
  50
  100
  150
  200
  250
  300
  350
  La ye
  rs R
  15 2
  R152
  where HSIC is the Hilbert-Schmidt independence criterion [15]. Given the centering matrix H = In− 1n11
  T and the centered Gram matrices K ′ = HKH and L′ = HLH , HSIC(K,L) = vec(K ′) · vec(L′)/(m− 1)2, the similarity between these centered Gram matrices. CKA is invariant to orthogonal transformation of representations (including permutation of neurons), and the normalization term ensures invariance to isotropic scaling. These properties enable meaningful comparison and analysis of neural network hidden representations. To work at scale with our models and tasks, we approximate the unbiased estimator of HSIC [39] using minibatches, as suggested in [29].
sections.heading: 4 Representation Structure of ViTs and Convolutional Networks
sections.text: We begin our investigation by using CKA to study the internal representation structure of each model. How are representations propagated within the two architectures, and are there signs of functional differences? To answer these questions, we take every pair of layers X,Y within a model and compute their CKA similarity. Note that we take representations not only from outputs of ViT/ResNet blocks, but also from intermediate layers, such as normalization layers and the hidden activations inside a ViT MLP. Figure 1 shows the results as a heatmap, for multiple ViTs and ResNets. We observe clear differences between the internal representation structure between the two model architectures: (1) ViTs show a much more uniform similarity structure, with a clear grid like structure (2) lower and higher layers in ViT show much greater similarity than in the ResNet, where similarity is divided into different (lower/higher) stages.
  We also perform cross-model comparisons, where we take all layers X from ViT and compare to all layers Y from ResNet. We observe (Figure 2) that the lower half of 60 ResNet layers are similar to approximately the lowest quarter of ViT layers. In particular, many more lower layers in the ResNet are needed to compute similar representations to the lower layers of ViT. The top half of the ResNet is approximately similar to the next third of the ViT layers. The final third of ViT layers is less similar to
  all ResNet layers, likely because this set of layers mainly manipulates the CLS token representation, further studied in Section 6.
  Taken together, these results suggest that (i) ViT lower layers compute representations in a different way to lower layers in the ResNet, (ii) ViT also more strongly propagates representations between lower and higher layers (iii) the highest layers of ViT have quite different representations to ResNet.
sections.heading: 5 Local and Global Information in Layer Representations
sections.text: In the previous section, we observed much greater similarity between lower and higher layers in ViT, and we also saw that ResNet required more lower layers to compute similar representations to a smaller set of ViT lower layers. In this section, we explore one possible reason for this difference: the difference in the ability to incorporate global information between the two models. How much global information is aggregated by early self-attention layers in ViT? Are there noticeable resulting differences to the features of CNNs, which have fixed, local receptive fields in early layers? In studying these questions, we demonstrate the influence of global representations and a surprising connection between scale and self-attention distances.
  Analyzing Attention Distances: We start by analyzing ViT self-attention layers, which are the mechanism for ViT to aggregate information from other spatial locations, and structurally very different to the fixed receptive field sizes of CNNs. Each self-attention layer comprises multiple self-attention heads, and for each head we can compute the average distance between the query patch position and the locations it attends to. This reveals how much local vs global information each self-attention layer is aggregating for the representation. Specifically, we weight the pixel distances by the attention weights for each attention head and average over 5000 datapoints, with results shown in Figure 3. In agreement with Dosovitskiy et al. [14], we observe that even in the lowest layers of ViT, self-attention layers have a mix of local heads (small distances) and global heads (large distances). This is in contrast to CNNs, which are hardcoded to attend only locally in the lower layers. At higher layers, all self-attention heads are global.
  Interestingly, we see a clear effect of scale on attention. In Figure 4, we look at attention distances when training only on ImageNet (no large-scale pre-training), which leads to much lower performance in ViT-L/16 and ViT-H/14 [14]. Comparing to Figure 3, we see that with not enough data, ViT does not learn to attend locally in earlier layers. Together, this suggests that using local information early on for image tasks (which is hardcoded into CNN architectures) is important for strong performance.
  Does access to global information result in different features? The results of Figure 3 demonstrate that ViTs have access to more global information than CNNs in their lower layers. But does this result in different learned features? As an interventional test, we take subsets of the ViT attention heads from the first encoder block, ranging from the subset corresponding to the most local attention
  heads to a subset of the representation corresponding to the most global attention heads. We then compute CKA similarity between these subsets and the lower layer representations of ResNet.
  The results, shown in Figure 5, which plot the mean distance for each subset against CKA similarity, clearly show a monotonic decrease in similarity as mean attention distance grows, demonstrating that access to more global information also leads to quantitatively different features than computed by the local receptive fields in the lower layers of the ResNet.
  Effective Receptive Fields: We conclude by computing effective receptive fields [23] for both ResNets and ViTs, with results in Figure 6 and Appendix C. We observe that lower layer effective receptive fields for ViT are indeed larger than in ResNets, and while ResNet effective receptive fields grow gradually, ViT receptive fields become much more global midway through the network. ViT receptive fields also show strong dependence on their center patch due to their strong residual connections, studied in the next section. As we show in Appendix C, in attention sublayers, receptive fields taken before the residual connection show far less dependence on this central patch.
sections.heading: 6 Representation Propagation through Skip Connections
sections.text: The results of the previous section demonstrate that ViTs learn different representations to ResNets in lower layers due to access to global information, which explains some of the differences in represen-
  Attention 1 Attention 3 Attention 6 Attention 9 Attention 12 ViT-B/32
  Initial Conv Block 3 Block 7 Block 12 Block 16 ResNet-50
  Figure 6: ResNet effective receptive fields are highly local and grow gradually; ViT effective receptive fields shift from local to global. We measure the effective receptive field of different layers as the absolute value of the gradient of the center location of the feature map (taken after residual connections) with respect to the input. Results are averaged across all channels in each map for 32 randomlyselected images.
  0 2 4 6 8 Token index
  0
  2
  4
  6
  8
  10
  Bl oc
  k In
  de x
  Ratio of Norms (first few tokens)
  2
  4
  6
  8
  10
  12
  0 2 4 6 8 10 12 14 Block Index
  0
  2
  4
  6
  8
  10
  Av er
  ag e
  no rm
  o f r
  ep re
  se nt
  at io
  n
  Ratio of Norms (aggregated) Self-Attention MLP Self-Attention CLS MLP CLS ResNet50
  Figure 7: Most information in ViT passes through skip connections. Comparison of representation norms between the skip-connection (identity) and the long branch for ViT-B/16 trained on ImageNet and a ResNet. For ViT, we show the CLS token separately from the rest of the representation. (left) shows the ratios separated for the first few tokens (token 0 is CLS), (right) shows averages over all tokens.
  tation structure observed in Section 4. However, the highly uniform nature of ViT representations (Figure 1) also suggests lower representations are faithfully propagated to higher layers. But how does this happen? In this section, we explore the role of skip connections in representation propagation across ViTs and ResNets, discovering ViT skip connections are highly influential, with a clear phase transition from preserving the CLS (class) token representation (in lower layers) to spatial token representations (in higher layers).
  Like Transformers, ViTs contain skip (aka identity or shortcut) connections throughout, which are added on after the (i) self-attention layer, and (ii) MLP layer. To study their effect, we plot the norm ratio ||zi||/||f(zi)|| where zi is the hidden representation of the ith layer coming from the skip connection, and f(zi) is the transformation of zi from the long branch (i.e. MLP or self-attention.)
  The results are in Figure 7 (with additional cosine similarity analysis in Figure E.2.) The heatmap on the left shows ||zi||/||f(zi)|| for different token representations. We observe a striking phase transition: in the first half of the network, the CLS token (token 0) representation is primarily propagated by the skip connection branch (high norm ratio), while the spatial token representations have a large contribution coming from the long branch (lower norm ratio). Strikingly, in the second half of the network, this is reversed.
  The right pane, which has line plots of these norm ratios across ResNet50, the ViT CLS token and the ViT spatial tokens additionally demonstrates that skip connection is much more influential in ViT compared to ResNet: we observe much higher norm ratios for ViT throughout, along with the phase transition from CLS to spatial token propagation (shown for the MLP and self-attention layers.)
  ViT Representation Structure without Skip Connections: The norm ratio results strongly suggest that skip connections play a key role in the representational structure of ViT. To test this interventionally, we train ViT models with skip connections removed in block i for varying i, and plot the CKA representation heatmap. The results, in Figure 8, illustrate that removing the skip connections in a block partitions the layer representations on either side. (We note a performance drop of 4% when
  0 1 2 3 4 5 6 0 1 2 3 4 5 6
  Token at Location(0, 0)
  0 1 2 3 4 5 6 0 1 2 3 4 5 6
  Token at Location(2, 1)
  0 1 2 3 4 5 6 0 1 2 3 4 5 6
  Token at Location(4, 2)
  0 1 2 3 4 5 6 0 1 2 3 4 5 6
  Token at Location(4, 4) ViT-B/32 final block
  ViT-L/16 final block
  removing skip connections from middle blocks.) This demonstrates the importance of representations being propagated by skip connections for the uniform similarity structure of ViT in Figure 1.
sections.heading: 7 Spatial Information and Localization
sections.text: The results so far, on the role of self-attention in aggregating spatial information in ViTs, and skipconnections faithfully propagating representations to higher layers, suggest an important followup question: how well can ViTs perform spatial localization? Specifically, is spatial information from the input preserved in the higher layers of ViT? And how does it compare in this aspect to ResNet? An affirmative answer to this is crucial for uses of ViT beyond classification, such as object detection.
  We begin by comparing token representations in the higher layers of ViT and ResNet to those of input patches. Recall that ViT tokens have a corresponding input patch, and thus a corresponding input spatial location. For ResNet, we define a token representation to be all the convolutional channels at a particular spatial location. This also gives it a corresponding input spatial location. We can then take a token representation and compute its CKA score with input image patches at different locations. The results are illustrated for different tokens (with their spatial locations labelled) in Figure 9.
  For ViT, we observe that tokens corresponding to locations at the edge of the image are similar to edge image patches, but tokens corresponding to interior locations are well localized, with their representations being most similar to the corresponding image patch. By contrast, for ResNet, we see significantly weaker localization (though Figure D.3 shows improvements for earlier layers.)
  One factor influencing this clear difference between architectures is that ResNet is trained to classify with a global average pooling step, while ViT has a separate classification (CLS) token. To examine this further, we test a ViT architecture trained with global average pooling (GAP) for localization (see Appendix A for training details). The results, shown in Figure 10, demonstrate that global average pooling does indeed reduce localization in the higher layers. More results in Appendix Section D.
  Localization and Linear Probe Classification: The previous results have looked at localization through direct comparison of each token with input patches. To complete the picture, we look at using each token separately to perform classification with linear probes. We do this across different layers of the model, training linear probes to classify image label with closed-form few-shot linear regression similar to Dosovitskiy et al. [14] (details in Appendix A). Results are in Figure 11, with further results in Appendix F. The left pane shows average accuracy of classifiers trained on individual tokens, where we see that ResNet50 and ViT with GAP model tokens perform well at higher layers, while in the standard ViT trained with a CLS token the spatial tokens do poorly – likely because their representations remain spatially localized at higher layers, which makes global classification challenging. Supporting this are results on the right pane, which shows that a single token from the
  ViT-GAP model achieves comparable accuracy in the highest layer to all tokens pooled together. With the results of Figure 9, this suggests all higher layer tokens in GAP models learn similar (global) representations.
sections.heading: 8 Effects of Scale on Transfer Learning
sections.text: Motivated by the results of Dosovitskiy et al. [14] that demonstrate the importance of dataset scale for high performing ViTs, and our earlier result (Figure 4) on needing scale for local attention, we perform a study of the effect of dataset scale on representations in transfer learning.
  We begin by studying the effect on representations as the JFT-300M pretraining dataset size is varied. Figure 12 illustrates the results on ViT-B/32 and ViT-L/16. Even with 3% of the entire dataset, lower layer representations are very similar to the model trained on the whole dataset, but higher layers require larger amounts of pretraining data to learn the same representations as at large data scale, especially with the large model size. In Section G, we study how much representations change in finetuning, finding heterogeneity over datasets.
  We next look at dataset size effect on the larger ViT-L/16 and ViT-H/14 models. Specifically, in the left pane of Figure 13, we train linear classifer probes on ImageNet classes for models pretrained on JFT-300M vs models only pretrained on ImageNet. We observe the JFT-300M pretained models achieve much higher accuracies even with middle layer representations, with a 30% gap in absolute
  accuracy to the models pretrained only on ImageNet. This suggests that for larger models, the larger dataset is especially helpful in learning high quality intermediate representations. This conclusion is further supported by the results of the right pane of Figure 13, which shows linear probes on different ResNet and ViT models, all pretrained on JFT-300M. We again see the larger ViT models learn much stronger intermediate representations than the ResNets. Additional linear probes experiments in Section F demonstrate this same conclusion for transfer to CIFAR-10 and CIFAR-100.
sections.heading: 9 Discussion
sections.text: Limitations: Our study uses CKA [17], which summarizes measurements into a single scalar, to provide quantitative insights on representation similarity. While we have complemented this with interventional tests and other analyses (e.g. linear probes), more fine-grained methods may reveal additional insights and variations in the representations.
  Conclusion: Given the central role of convolutional neural networks in computer vision breakthroughs, it is remarkable that Transformer architectures (almost identical to those used in language) are capable of similar performance. This raises fundamental questions on whether these architectures work in the same way as CNNs. Drawing on representational similarity techniques, we find surprisingly clear differences in the features and internal structures of ViTs and CNNs. An analysis of self-attention and the strength of skip connections demonstrates the role of earlier global features and strong representation propagation in ViTs for these differences, while also revealing that some CNN properties, e.g. local information aggregation at lower layers, are important to ViTs, being learned from scratch at scale. We examine the potential for ViTs to be used beyond classification through a study of spatial localization, discovering ViTs with CLS tokens show strong preservation of spatial information — promising for future uses in object detection. Finally, we investigate the effect of scale for transfer learning, finding larger ViT models develop significantly stronger intermediate representations through larger pretraining datasets. These results are also very pertinent to understanding MLP-based architectures for vision proposed by concurrent work [42, 43], further discussed in Section H, and together answer central questions on differences between ViTs and CNNs, and suggest new directions for future study. From the perspective of societal impact, these findings and future work may help identify potential failures as well as greater model interpretability.
title: Do Vision Transformers See Like Convolutional Neural Networks?