summaries: The paper propose to inject inductive bias (IB) for vision tasks to boost the performance of vision transformers (ViT). The main contributions of the paper are two-fold: a reduction cell that aggregate the features from different scales by controlling the dilation of a  few of parallel convolution branches and a parallel convolution branch with the multi-head self-attention (MHSA) block. The results on ImageNet is promising compared with the selected baselines but I found it is hard to get a conclusive result on downstream classification tasks.

summaries: This paper proposed two types of basic cells to modify the vanilla vision transformer structure: reduction cell (RC) and normal cell (NC). RCs are used to down-sample and embed the input images into tokens and NCs are to jointly model locality and global dependencies in the token sequence. Specifically, they used paralleled attention module and convolutional layers followed by a feed-forward network (FFN) to construct these two cells. Experiments on ImageNet and downstream tasks verified the superiority of the proposed network design.

summaries: The standard vision transformer model lacks the capability to model inductive biases desirable for images e.g., local features encoding, and scale invariance. It, therefore, requires large-scale training data (300 million images), or other variants (e.g., Data Efficient ViT) distilling Knowledge from a pertained CNN. The paper proposes an architecture, which tries to take care of these inductive biases. The architecture comprises of 3 “Reduction cells” at the beginning followed by multiple “Normal Cells”. A reduction cell gets a pyramid effect using dilated convolutions at the beginning which gradually downsamples features. The normal cell doesn’t have any dilated convolutions. Each block has Convolution and attention running in parallel.  The assumption here is that convolution captures local relationships, pyramid structure (via diluted convolutions) captures scale invariance, and attention captures global relationships.

summaries: A new vision transformer architecture VITAE, with a better use of intrinsic inductive bias IB via a new architecture using convolutional dillated layers. Both scale and locality IB are enabled by that architecture. Empirical results show that the new method can achieve excellent results with much less parameters than other recent methods.

summaries: This work incorporates two architecture changes which induce the inductive biases of 1) locality and 2) scale-invariance. It does so though a sequence of Reduction Cells followed by Normal Cells. In doing so, the authors demonstrate comparable performance to strong CNN baselines and improvements over Transformer-based ImageNet models -- the top performing ViTAE model achieves a 83.0% Top-1 accuracy.

abstractText: Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we propose a new Vision Transformer Advanced by Exploring intrinsic IB from convolutions, i.e., ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block in parallel to the multi-head selfattention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of ViTAE over the baseline transformer and concurrent works. Source code and pretrained models will be available at code.

authors:
E.H. Adelson
C.H. Anderson
J.R. Bergen
P.J. Burt
J.M. Ogden

title: Pyramid methods in image processing

venue: RCA engineer, 29(6):33–41

authors:
J.L. Ba
J.R. Kiros
G.E. Hinton

title: Layer normalization

venue: arXiv preprint arXiv:1607.06450

authors:
H. Bay
T. Tuytelaars
L. Van Gool

title: Surf: Speeded up robust features

venue: European conference on computer vision, pages 404–417. Springer

authors:
P.J. Burt
E.H. Adelson

title: The laplacian pyramid as a compact image code

venue: Readings in computer vision, pages 671–679. Elsevier

authors:
Z. Cai
N. Vasconcelos

title: Cascade r-cnn: Delving into high quality object detection

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154–6162

authors:
C.-F. Chen
Q. Fan
R. Panda

title: Crossvit: Cross-attention multi-scale vision transformer for image classification

venue: arXiv preprint arXiv:2103.14899

authors:
H. Chen
Y. Wang
T. Guo
C. Xu
Y. Deng
Z. Liu
S. Ma
C. Xu
C. Xu
W. Gao

title: Pre-trained image processing transformer

venue: arXiv preprint arXiv:2012.00364

authors:
L.-C. Chen
G. Papandreou
F. Schroff
H. Adam

title: Rethinking atrous convolution for semantic image segmentation

venue: arXiv preprint arXiv:1706.05587

authors:
X. Chen
S. Xie
K. He

title: An empirical study of training self-supervised vision transformers

venue: arXiv preprint arXiv:2104.02057

authors:
X. Chu
Z. Tian
Y. Wang
B. Zhang
H. Ren
X. Wei
H. Xia
C. Shen

title: Twins: Revisiting spatial attention design in vision transformers

venue: arXiv preprint arXiv:2104.13840

authors:
X. Chu
Z. Tian
B. Zhang
X. Wang
X. Wei
H. Xia
C. Shen

title: Conditional positional encodings for vision transformers

venue: arXiv preprint arXiv:2102.10882

title: MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark

title: Openmmlab pose estimation toolbox and benchmark

venue: https://github.com/ open-mmlab/mmpose

authors:
Z. Dai
Z. Yang
Y. Yang
J. Carbonell
Q.V. Le
R. Salakhutdinov

title: Transformer-xl: Attentive language models beyond a fixed-length context

venue: arXiv preprint arXiv:1901.02860

authors:
S. d’Ascoli
H. Touvron
M. Leavitt
A. Morcos
G. Biroli
L. Sagun

title: Convit: Improving vision transformers with soft convolutional inductive biases

venue: arXiv preprint arXiv:2103.10697,

authors:
H. Demirel
G. Anbarjafari

title: Image resolution enhancement by using discrete and stationary wavelet decomposition

venue: IEEE transactions on image processing, 20(5):1458–1460

authors:
J. Devlin
M.-W. Chang
K. Lee
K. Toutanova

title: Bert: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805

authors:
X. Ding
X. Zhang
J. Han
G. Ding

title: Repmlp: Re-parameterizing convolutions into fully-connected layers for image recognition

venue: arXiv preprint arXiv:2105.01883

authors:
A. Dosovitskiy
L. Beyer
A. Kolesnikov
D. Weissenborn
X. Zhai
T. Unterthiner
M. Dehghani
M. Minderer
G. Heigold
S. Gelly

title: et al

venue: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

authors:
A. El-Nouby
H. Touvron
M. Caron
P. Bojanowski
M. Douze
A. Joulin
I. Laptev
N. Neverova
G. Synnaeve
J. Verbeek

title: et al

venue: Xcit: Cross-covariance image transformers. arXiv preprint arXiv:2106.09681

authors:
B. Graham
A. El-Nouby
H. Touvron
P. Stock
A. Joulin
H. Jégou
M. Douze

title: Levit: a vision transformer in convnet’s clothing for faster inference

venue: arXiv preprint arXiv:2104.01136

authors:
M.-H. Guo
Z.-N. Liu
T.-J. Mu
S.-M. Hu

title: Beyond self-attention: External attention using two linear layers for visual tasks

venue: arXiv preprint arXiv:2105.02358

authors:
K. Han
A. Xiao
E. Wu
J. Guo
C. Xu
Y. Wang

title: Transformer in transformer

venue: arXiv preprint arXiv:2103.00112

authors:
K. He
G. Gkioxari
P. Dollár
R. Girshick

title: Mask r-cnn

venue: Proceedings of the IEEE international conference on computer vision, pages 2961–2969

authors:
K. He
X. Zhang
S. Ren
J. Sun

title: Spatial pyramid pooling in deep convolutional networks for visual recognition

venue: IEEE transactions on pattern analysis and machine intelligence, 37(9):1904–1916

title: Deep residual learning for image recognition

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778

authors:
L. He
Y. Dong
Y. Wang
D. Tao
Z. Lin

title: Gauge equivariant transformer

venue: Thirty-Fifth Conference on Neural Information Processing Systems

authors:
B. Heo
S. Yun
D. Han
S. Chun
J. Choe
S.J. Oh

title: Rethinking spatial dimensions of vision transformers

venue: International Conference on Computer Vision (ICCV)

venue: arXiv preprint arXiv:2103.16302

authors:
G. Hinton
O. Vinyals
J. Dean

title: Distilling the knowledge in a neural network

venue: NIPS Deep Learning and Representation Learning Workshop

authors:
A.G. Howard
M. Zhu
B. Chen
D. Kalenichenko
W. Wang
T. Weyand
M. Andreetto
H. Adam

title: Mobilenets: Efficient convolutional neural networks for mobile vision applications

venue: arXiv preprint arXiv:1704.04861

authors:
G. Huang
Z. Liu
L. Van Der Maaten
K.Q. Weinberger

title: Densely connected convolutional networks

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Z. Huang
Y. Ben
G. Luo
P. Cheng
G. Yu
B. Fu

title: Shuffle transformer: Rethinking spatial shuffle for vision transformer

venue: arXiv preprint arXiv:2106.03650

authors:
S. Ioffe
C. Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

venue: International conference on machine learning, pages 448–456. PMLR

authors:
Y. Ke
R. Sukthankar

title: Pca-sift: A more distinctive representation for local image descriptors

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, volume 2, pages II–II. IEEE

authors:
J. Krause
M. Stark
J. Deng
L. Fei-Fei

title: 3d object representations for fine-grained categorization

venue: 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia

authors:
A. Krizhevsky
G. Hinton

title: Learning multiple layers of features from tiny images

authors:
A. Krizhevsky
I. Sutskever
G.E. Hinton

title: Imagenet classification with deep convolutional neural networks

venue: Advances in neural information processing systems, 25:1097–1105

authors:
W.-S. Lai
J.-B. Huang
N. Ahuja
M.-H. Yang

title: Deep laplacian pyramid networks for fast and accurate super-resolution

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 624–632

authors:
Z. Lan
M. Chen
S. Goodman
K. Gimpel
P. Sharma
R. Soricut

title: Albert: A lite bert for self-supervised learning of language representations

venue: arXiv preprint arXiv:1909.11942

authors:
Y. LeCun
Y. Bengio

title: et al

venue: Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995

authors:
Y. LeCun
Y. Bengio
G. Hinton

title: Deep learning

authors:
Y. Li
K. Zhang
J. Cao
R. Timofte
L. Van Gool

title: Localvit: Bringing locality to vision transformers

venue: arXiv preprint arXiv:2104.05707

authors:
G. Lin
C. Shen

title: A

venue: Van Den Hengel, and I. Reid. Efficient piecewise training of deep structured models for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3194–3203

authors:
T.-Y. Lin
P. Dollár
R. Girshick
K. He
B. Hariharan
S. Belongie

title: Feature pyramid networks for object detection

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125

authors:
Y. Liu
M. Ott
N. Goyal
J. Du
M. Joshi
D. Chen
O. Levy
M. Lewis
L. Zettlemoyer
V. Stoyanov

title: Roberta: A robustly optimized bert pretraining approach

venue: arXiv preprint arXiv:1907.11692

authors:
Z. Liu
Y. Lin
Y. Cao
H. Hu
Y. Wei
Z. Zhang
S. Lin
B. Guo

title: Swin transformer: Hierarchical vision transformer using shifted windows

venue: arXiv preprint arXiv:2103.14030

authors:
I. Loshchilov
F. Hutter

title: Decoupled weight decay regularization

venue: International Conference on Learning Representations

authors:
W. Luo
Y. Li
R. Urtasun
R.S. Zemel

title: Understanding the effective receptive field in deep convolutional neural networks

venue: Proceedings of the 30th International Conference on Neural Information Processing Systems, volume 29, pages 4898–4906

title: Do you even need attention? a stack of feed-forward layers does surprisingly well on imagenet

venue: arXiv: Computer Vision and Pattern Recognition

authors:
H. Nam
J.-W. Ha
J. Kim

title: Dual attention networks for multimodal reasoning and matching

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 299–307

authors:
P.C. Ng
S. Henikoff

title: Sift: Predicting amino acid changes that affect protein function

venue: Nucleic acids research, 31(13):3812–3814

authors:
M.-E. Nilsback
A. Zisserman

title: Automated flower classification over a large number of classes

venue: In Indian Conference on Computer Vision, Graphics and Image Processing,

authors:
S.W. Oh
J.-Y. Lee
N. Xu
S.J. Kim

title: Video object segmentation using space-time memory networks

venue: Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9226–9235

authors:
H. Olkkonen
P. Pesola

title: Gaussian pyramid wavelet transform for multiresolution analysis of images

venue: Graphical Models and Image Processing, 58(4):394–398

authors:
O.M. Parkhi
A. Vedaldi
A. Zisserman
C.V. Jawahar

title: Cats and dogs

venue: IEEE Conference on Computer Vision and Pattern Recognition

authors:
A. Paszke
S. Gross
F. Massa
A. Lerer
J. Bradbury
G. Chanan
T. Killeen
Z. Lin
N. Gimelshein
L. Antiga
A. Desmaison
A. Kopf
E. Yang
Z. DeVito
M. Raison
A. Tejani
S. Chilamkurthy
B. Steiner
L. Fang
J. Bai
S. Chintala

title: Pytorch: An imperative style

venue: high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, pages 8026–8037

authors:
Z. Peng
W. Huang
S. Gu
L. Xie
Y. Wang
J. Jiao
Q. Ye

title: Conformer: Local features coupling global representations for visual recognition

venue: arXiv preprint arXiv:2105.03889

authors:
F. Perazzi
J. Pont-Tuset
B. McWilliams
L. Van Gool
M. Gross
A. Sorkine-Hornung

title: A benchmark dataset and evaluation methodology for video object segmentation

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724–732

authors:
J. Pont-Tuset
F. Perazzi
S. Caelles
P. Arbeláez
A. Sorkine-Hornung
L. Van Gool

title: The 2017 davis challenge on video object segmentation

venue: arXiv preprint arXiv:1704.00675

authors:
A. Radford
J. Wu
R. Child
D. Luan
D. Amodei
I. Sutskever

title: Language models are unsupervised multitask learners

venue: OpenAI blog, 1(8):9

authors:
I. Radosavovic
R.P. Kosaraju
R. Girshick
K. He
P. Dollár

title: Designing network design spaces

authors:
E. Rublee
V. Rabaud
K. Konolige
G. Bradski

title: Orb: An efficient alternative to sift or surf

venue: Proceedings of the IEEE international conference on computer vision, pages 2564–2571. Ieee

authors:
S. Sabour
N. Frosst
G.E. Hinton

title: Dynamic routing between capsules

venue: arXiv preprint arXiv:1710.09829

authors:
M. Sandler
A. Howard
M. Zhu
A. Zhmoginov
L.-C. Chen

title: Mobilenetv2: Inverted residuals and linear bottlenecks

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520

authors:
R.R. Selvaraju
M. Cogswell
A. Das
R. Vedantam
D. Parikh
D. Batra

title: Grad-cam: Visual explanations from deep networks via gradient-based localization

venue: Proceedings of the IEEE international conference on computer vision, pages 618–626

authors:
P. Shaw
J. Uszkoreit
A. Vaswani

title: Self-attention with relative position representations

venue: arXiv preprint arXiv:1803.02155

authors:
K. Simonyan
A. Zisserman

title: Very deep convolutional networks for large-scale image recognition

venue: arXiv preprint arXiv:1409.1556

authors:
A. Srinivas
T.-Y. Lin
N. Parmar
J. Shlens
P. Abbeel
A. Vaswani

title: Bottleneck transformers for visual recognition

venue: arXiv preprint arXiv:2101.11605

authors:
C. Szegedy
S. Ioffe
V. Vanhoucke
A. Alemi

title: Inception-v4

venue: inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31

authors:
C. Szegedy
W. Liu
Y. Jia
P. Sermanet
S. Reed
D. Anguelov
D. Erhan
V. Vanhoucke
A. Rabinovich

title: Going deeper with convolutions

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9

authors:
C. Szegedy
V. Vanhoucke
S. Ioffe
J. Shlens
Z. Wojna

title: Rethinking the inception architecture for computer vision

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826

authors:
M. Tan
Q. Le

title: Efficientnet: Rethinking model scaling for convolutional neural networks

venue: International Conference on Machine Learning, pages 6105–6114. PMLR

authors:
I. Tolstikhin
N. Houlsby
A. Kolesnikov
L. Beyer
X. Zhai
T. Unterthiner
J. Yung
D. Keysers
J. Uszkoreit
M. Lucic
A. Dosovitskiy

title: Mlp-mixer: An all-mlp architecture for vision

venue: arXiv preprint arXiv:2105.01601

authors:
H. Touvron
P. Bojanowski
M. Caron
M. Cord
A. El-Nouby
E. Grave
A. Joulin
G. Synnaeve
J. Verbeek
H. Jégou

title: Resmlp: Feedforward networks for image classification with data-efficient training

venue: arXiv preprint arXiv:2105.03404

authors:
H. Touvron
M. Cord
M. Douze
F. Massa
A. Sablayrolles
H. Jégou

title: Training data-efficient image transformers & distillation through attention

venue: arXiv preprint arXiv:2012.12877

authors:
H. Touvron
M. Cord
A. Sablayrolles
G. Synnaeve
H. Jégou

title: Going deeper with image transformers

venue: arXiv preprint arXiv:2103.17239

authors:
H. Touvron
A. Sablayrolles
M. Douze
M. Cord
H. Jégou

title: Grafit: Learning fine-grained image representations with coarse labels

venue: arXiv preprint arXiv:2011.12982

authors:
A. Vaswani
N. Shazeer
N. Parmar
J. Uszkoreit
L. Jones
A.N. Gomez
L. Kaiser
I. Polosukhin

title: Attention is all you need

venue: Proceedings of the 31st International Conference on Neural Information Processing Systems, volume 30, pages 5998–6008

authors:
W. Wang
E. Xie
X. Li
D.-P. Fan
K. Song
D. Liang
T. Lu
P. Luo
L. Shao

title: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

venue: arXiv preprint arXiv:2102.12122

authors:
X. Wang
R. Girshick
A. Gupta
K. He

title: Non-local neural networks

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803

title: Pytorch image models

authors:
H. Wu
B. Xiao
N. Codella
M. Liu
X. Dai
L. Yuan
L. Zhang

title: Cvt: Introducing convolutions to vision transformers

venue: arXiv preprint arXiv:2103.15808

authors:
B. Xiao
H. Wu
Y. Wei

title: Simple baselines for human pose estimation and tracking

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
T. Xiao
Y. Liu
B. Zhou
Y. Jiang
J. Sun

title: Unified perceptual parsing for scene understanding

venue: Proceedings of the European Conference on Computer Vision (ECCV), pages 418–434

authors:
J. Xie
R. Zeng
Q. Wang
Z. Zhou
P. Li

title: So-vit: Mind visual tokens for vision transformer

venue: arXiv preprint arXiv:2104.10935

authors:
S. Xie
R. Girshick
P. Dollár
Z. Tu
K. He

title: Aggregated residual transformations for deep neural networks

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500

authors:
W. Xu
Y. Xu
T. Chang
Z. Tu

title: Co-scale conv-attentional image transformers

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),

authors:
H. Yan
Z. Li
W. Li
C. Wang
M. Wu

title: and C

venue: Zhang. Contnet: Why not use convolution and transformer at the same time? arXiv preprint arXiv:2104.13497

authors:
F. Yu
V. Koltun

title: Multi-scale context aggregation by dilated convolutions

venue: ICLR 2016 : International Conference on Learning Representations 2016

authors:
F. Yu
V. Koltun
T. Funkhouser

title: Dilated residual networks

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 472–480

authors:
K. Yuan
S. Guo
Z. Liu
A. Zhou
F. Yu
W. Wu

title: Incorporating convolution designs into visual transformers

venue: arXiv preprint arXiv:2103.11816

authors:
L. Yuan
Y. Chen
T. Wang
W. Yu
Y. Shi
Z. Jiang
F.E. Tay
J. Feng
S. Yan

title: Tokens-to-token vit: Training vision transformers from scratch on imagenet

venue: arXiv preprint arXiv:2101.11986

authors:
M.D. Zeiler
R. Fergus

title: Visualizing and understanding convolutional networks

venue: European conference on computer vision, pages 818–833. Springer

authors:
X. Zhang
X. Zhou
M. Lin
J. Sun

title: Shufflenet: An extremely efficient convolutional neural network for mobile devices

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856

authors:
H. Zhao
J. Shi
X. Qi
X. Wang
J. Jia

title: Pyramid scene parsing network

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890

authors:
S. Zheng
J. Lu
H. Zhao
X. Zhu
Z. Luo
Y. Wang
Y. Fu
J. Feng
T. Xiang
P.H.S. Torr
L. Zhang

title: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers

venue: arXiv preprint arXiv:2012.15840

authors:
B. Zhou
H. Zhao
X. Puig
S. Fidler
A. Barriuso
A. Torralba

title: Scene parsing through ade20k dataset

venue: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641

authors:
B. Zhou
H. Zhao
X. Puig
T. Xiao
S. Fidler
A. Barriuso
A. Torralba

title: Semantic understanding of scenes through the ade20k dataset

venue: International Journal of Computer Vision, 127(3):302–321

authors:
X. Zhu
W. Su
L. Lu
B. Li
X. Wang
J. Dai

title: Deformable detr: Deformable transformers for end-to-end object detection

text:
1 Introduction
Transformers [79, 17, 40, 14, 46, 61] have shown a domination trend in NLP studies owing to their strong ability in modeling long-range dependencies by the self-attention mechanism [67, 81, 51]. Such success and good properties of transformers has inspired following many works that apply them in various computer vision tasks [19, 100, 97, 80, 7]. Among them, ViT [19] is the pioneering pure transformer model that embeds images into a sequence of visual tokens and models the global dependencies among them with stacked transformer blocks. Although it achieves promising performance on image classification, it requires large-scale training data and a longer training schedule. One important reason is that ViT
∗Equal Contribution. Interns at JD Explore Academy.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
lacks intrinsic inductive bias (IB) in modeling local visual structures (e.g., edges and corners) and dealing with objects at various scales like convolutions. Alternatively, ViT has to learn such IB implicitly from large-scale data.
Unlike vision transformers, Convolution Neural Networks (CNNs) naturally equip with the intrinsic IBs of scale-invariance and locality and still serve as prevalent backbones in vision tasks [26, 70, 62, 8, 96]. The success of CNNs inspires us to explore intrinsic IBs in vision transformers. We start by analyzing the above two IBs of CNNs, i.e., locality and scale-invariance. Convolution that computes local correlation among neighbor pixels is good at extracting local features such as edges and corners. Consequently, CNNs can provide plentiful low-level features at the shallow layers [94], which are then aggregated into high-level features progressively by a bulk of sequential convolutions [32, 68, 71]. Moreover, CNNs have a hierarchy structure to extract multi-scale features at different layers [68, 38, 26]. Besides, intra-layer convolutions can also learn features at different scales by varying their kernel sizes and dilation rates [25, 70, 8, 45, 96]. Consequently, scale-invariant feature representation can be obtained via intra- or inter-layer feature fusion. Nevertheless, CNNs are not well suited to model long-range dependencies2, which is the key advantage of transformers. An interesting question comes up: Can we improve vision transformers by leveraging the good properties of CNNs? Recently, DeiT [76] explores the idea of distilling knowledge from CNNs to transformers to facilitate training and improve the performance. However, it requires an off-the-shelf CNN model as the teacher and consumes extra training cost.
Different from DeiT, we explicitly introduce intrinsic IBs into vision transformers by re-designing the network structures in this paper. Current vision transformers always obtain tokens with singlescale context [19, 93, 80, 86, 47, 69, 77] and learn to adapt to objects at different scales from data. For example, T2T-ViT [93] improves ViT by delicately generating tokens in a soft split manner. Specifically, it uses a series of Tokens-to-Token transformation layers to aggregate single-scale neighboring contextual information and progressively structurizes the image to tokens. Motivated by the success of CNNs in dealing with scale variance, we explore a similar design in transformers, i.e., intra-layer convolutions with different receptive fields [70, 91], to embed multi-scale context into tokens. Such a design allows tokens to carry useful features of objects at various scales, thereby naturally having the intrinsic scale-invariance IB and explicitly facilitating transformers to learn scale-invariant features more efficiently from data. On the other hand, low-level local features are fundamental elements to generate high-level discriminative features. Although transformers can also learn such features at shallow layers from data, they are not skilled as convolutions by design. Recently, [89, 43, 21] stack convolutions and attention layers sequentially and demonstrate that locality is a reasonable compensation of global dependency. However, this serial structure ignores the global context during locality modeling (and vice versa). To avoid such a dilemma, we follow the “divide-and-conquer” idea and propose to model locality and long-range dependencies in parallel and then fuse the features to account for both. In this way, we empower transformers to learn local and long-range features within each block more effectively.
Technically, we propose a new Vision Transformers Advanced by Exploring Intrinsic Inductive Bias (ViTAE), which is a combination of two types of basic cells, i.e., reduction cell (RC) and normal cell (NC). RCs are used to downsample and embed the input images into tokens with rich multi-scale context while NCs aim to jointly model locality and global dependencies in the token sequence. Moreover, these two types of cells share a simple basic structure, i.e., paralleled attention module and convolutional layers followed by a feed-forward network (FFN). It is noteworthy that RC has an extra pyramid reduction module with atrous convolutions of different dilation rates to embed multi-scale context into tokens. Following the setting in [93], we stack three reduction cells to reduce the spatial resolution by 1/16 and a series of NCs to learn discriminative features from data. ViTAE outperforms representative vision transformers in terms of data efficiency and training efficiency (see Figure 1), as well as classification accuracy and generalization on downstream tasks.
Our contributions are threefold. First, we explore two types of intrinsic IB in transformers, i.e., scale invariance and locality, and demonstrate the effectiveness of this idea in improving the feature learning ability of transformers. Second, we design a new transformer architecture named ViTAE based on two new reduction and normal cells to intrinsically incorporate the above two IBs. The proposed ViTAE embeds multi-scale context into tokens and learns both local and long-range features
2Despite projection in transformer can be viewed as 1 × 1 convolution [9], the term of convolution here refers to those with larger kernels, e.g., 3× 3, which are widely used in typical CNNs to extract spatial features.
effectively. Third, ViTAE outperforms representative vision transformers regarding classification accuracy, data efficiency, training efficiency, and generalization on downstream tasks. ViTAE achieves 75.3% and 82.0% top-1 accuracy on ImageNet with 4.8M and 23.6M parameters, respectively.

heading: 2 Related Work

heading: 2.1 CNNs with intrinsic IB

text: CNNs have led to a series of breakthroughs in image classification [38, 94, 26, 95, 87] and downstream computer vision tasks. The convolution operations in CNNs extract local features from the neighbor pixels within the receptive field determined by the kernel size [42]. Following the intuition that local pixels are more likely to be correlated in images [41], CNNs have the intrinsic IB in modeling locality. In addition to the locality, another critical topic in visual tasks is scale-invariance, where multi-scale features are needed to represent the objects at different scales effectively [49, 90]. For example, to effectively learn features of large objects, a large receptive field is needed by either using large convolution kernels [90, 91] or a series of convolution layers in deeper architectures [26, 32, 68, 71]. To construct multi-scale feature representation, the classical idea is using image pyramid [8, 1, 55, 4, 39, 16], where features are hand-crafted or learned from a pyramid of images at different resolutions respectively [44, 8, 52, 63, 35, 3]. Accordingly, features from the small scale image mainly encode the large objects while features from the large scale image respond more to small objects. In addition to the above inter-layer fusion way, another way is to aggregate multi-scale context by using multiple convolutions with different receptive fields within a single layer, i.e., intra-layer fusion [96, 71, 70, 70, 72]. Either inter-layer fusion or intra-layer fusion empower CNNs an intrinsic IB in modeling scale-invariance. This paper introduces such an IB to vision transformers by following the intra-layer fusion idea and utilizing multiple convolutions with different dilation rates in the reduction cells to encode multi-scale context into each visual token.

heading: 2.2 Vision transformers with learned IB

text: ViT [19] is the pioneering work that applies a pure transformer to vision tasks and achieves promising results. However, since ViT lacks intrinsic inductive bias in modeling local visual structures, it indeed learns the IB from amounts of data implicitly. Following works along this direction are to simplify the model structures with fewer intrinsic IBs and directly learn them from large scale data [50, 74, 75, 22, 18, 20, 27] which have achieved promising results and been studied actively. Another direction is to leverage the intrinsic IB from CNNs to facilitate the training of vision transformers, e.g., using less training data or shorter training schedules. For example, DeiT [76] proposes to distill knowledge from CNNs to transformers during training. However, it requires an off-the-shelf CNN model as a teacher, introducing extra computation cost during training. Recently, some works try to introduce the intrinsic IB of CNNs into vision transformers explicitly [23, 58, 21, 43, 15, 89, 83, 92, 6, 47, 11]. For example, [43, 21, 83] stack convolutions and attention layers sequentially, resulting in a serial structure and modeling the locality and global dependency accordingly. [80, 28] design sequential stage-wise structures while [47, 33] apply attention within local windows. However, these serial structure may ignore the global context during locality modeling (and vice versa). [88] establishes connection across different scales at the cost of heavy computation. Instead, we follow the “divide-and-conquer” idea and propose to model locality and global dependencies simultaneously via a parallel structure within each transformer layer. Conformer [58], the most relevant concurrent work to us, employs a unit to explore inter-block interactions between parallel convolution and transformer blocks. In contrast, in ViTAE, the convolution and attention modules are designed to be complementary to each other within the transformer block. In addition, Conformer is not designed to have inherent scale invariance IB.

heading: 3 Methodology

heading: 3.1 Revisit vision transformer

text:
We first give a brief review of vision transformer in this part. To adapt transformers to vision tasks, ViT [19] first splits an image x ∈ RH×W×C into tokens with a reduction ratio of p (i.e., xt ∈ R((H×W )/p 2)×D), where H , W and C denote the height, width, and channel dimensions of
the input image, D = Cp2 denotes the token dimension. Then, an extra class token is concatenated to the visual tokens before adding position embeddings in an element-wise manner. The resulting tokens are fed into the following transformer layers. Each transformer layer is composed of two parts, i.e., a multi-head self-attention module (MHSA) and a feed forward network (FFN).
MHSA Multi-head self-attention extends single-head self-attention (SHSA) by using different projection matrices for each head. Specifically, the input tokens xt are first projected to queries (Q), keys (K) and values (V ) using projection matrices, i.e., Q,K, V = xtWQ, xtQK , xtQV , where WQ/K/V ∈ RD×D denotes the projection matrix for query, key, and value, respectively. Then, the self-attention operation is calculated as:
Attention(Q,K, V ) = softmax( QKT√
D )V. (1)
This SHSA module is repeated for h times to formulate the MHSA module, where h is the number of heads. The output features of the h heads are concatenated along the channel dimension and formulate the output of the MHSA module.
FFN FFN is placed on top of the MHSA module and applied to each token identically and separately. It consists of two linear transformations with an activation function in between. Besides, a layer normalization [2] and a shortcut are added before and aside from the MHSA and FFN, respectively.

heading: 3.2 Overview architecture of ViTAE

text: ViTAE aims to introduce the intrinsic IB in CNNs to vision transformers. As shown in Figure 2, ViTAE is composed of two types of cells, i.e., RCs and NCs. RCs are responsible for embedding multi-scale context and local information into tokens, and NCs are used to further model the locality and long-range dependencies in the tokens. Taken an image x ∈ RH×W×C as input, three RCs are used to gradually downsample x by 4×, 2×, and 2×, respectively. Thereby, the output tokens of the RCs are of size [H/16,W/16, D] where D is the token dimension (64 in our experiments). The output tokens of RCs are then flattened as RHW/256×D, concatenated with the class token, and added by the sinusoid position encoding. Next, the tokens are fed into the following NCs, which keep the length of the tokens. Finally, the prediction probability is obtained using a linear classification layer on the class token from the last NC.

heading: 3.3 Reduction cell

text:
Instead of directly splitting and flatten images into visual tokens based on a linear image patch embedding layer, we devise the reduction cell to embed multi-scale context and local information into visual tokens, which introduces the intrinsic scale-invariance and locality IBs from convolutions. Technically, RC has two parallel branches responsible for modeling locality and long-range dependency, respectively, followed by an FFN for feature transformation. We denote the input feature of the ith RC as fi ∈ RHi×Wi×Di . The input of the first RC is the image x. In the global dependencies branch, fi is firstly fed into a Pyramid Reduction Module (PRM) to extract multi-scale context, i.e.,
fmsi , PRMi(fi) = Cat([Convij(fi; sij , ri)|sij ∈ Si, ri ∈ R]), (2)
where Convij(·) indicates the jth convolutional layer in the PRM (PRMi(·)). It uses a dilation rate sij from the predefined dilation rate set Si corresponding to the ith RC. Note that we use stride convolution to reduce the spatial dimension of features by a ratio ri from the predefined reduction ratio setR. The conv features are concatenated along the channel dimension, i.e., fmsi ∈ R(Wi/p)×(Hi/p)×(|Si|D), where |Si| denotes the number of dilation rates in Si. fmsi is then processed by an MHSA module to model long-range dependencies, i.e.,
fgi = MHSAi(Img2Seq(f ms i )), (3)
where Img2Seq(·) is a simple reshape operation to flatten the feature map to a 1D sequence. In this way, fgi embeds the multi-scale context in each token. In addition, we use a Parallel Convolutional Module (PCM) to embed local context within the tokens, which are fused with fgi as follows:
f lgi = f g i + PCMi(fi). (4)
Here, PCMi(·) represents the PCM, which is composed of three stacked convolution layers and an Img2Seq(·) operation. It is noteworthy that the parallel convolution branch has the same spatial downsampling ratio as the PRM by using stride convolutions. In this way, the token features can carry both local and multi-scale context, implying that RC acquires the locality IB and scale-invariance IB by design. The fused tokens are then processed by the FFN, reshaped back to feature maps, and fed into the following RC or NC, i.e.,
fi+1 = Seq2Img(FFNi(f lg i ) + f lg i ), (5)
where the Seq2Img(·) is a simple reshape operation to reshape a token sequence back to feature maps. FFNi(·) represents the FFN in the ith RC. In our ViTAE, three RCs are stacked sequentially to gradually reduce the input image’s spatial dimension by 4×, 2×, and 2×, respectively. The feature maps generated by the last RC are of a size of [H/16,W/16, D], which are then flattened into visual tokens and fed into the following NCs.

heading: 3.4 Normal cell

text:
As shown in the bottom right part of Figure 2, NCs share a similar structure with the reduction cell except for the absence of the PRM. Due to the relatively small ( 116×) spatial size of feature maps after RCs, it is unnecessary to use PRM in NCs. Given f3 from the third RC, we first concatenate it with the class token tcls, and then add it to the positional encodings to get the input tokens t for the following NCs. Here we ignore the subscript for clarity since all NCs have an identical architecture but different learnable weights. tcls is randomly initialized at the start of training and fixed during the inference. Similar to the RC, the tokens are fed into the MHSA module, i.e., tg = MHSA(t). Meanwhile, they are reshaped to 2D feature maps and fed into the PCM, i.e., tl = Img2Seq(PCM(Seq2Img(t))). Note that the class token is discarded in PCM because it has no spatial connections with other visual tokens. To further reduce the parameters in NCs, we use group convolutions in PCM. The features from MHSA and PCM are then fused via element-wise sum, i.e., tlg = tg + tl. Finally, tlg are fed into the FFN to get the output features of NC, i.e., tnc = FFN(tlg) + tlg. Similar to ViT [19], we apply layer normalization to the class token generated by the last NC and feed it to the classification head to get the final classification result.
3.5 Model details
We use two variants of ViTAE in our experiments for a fair comparison of other models with similar model sizes. The details of them are summarized in Table 1. In the first RC, the default convolution kernel size is 7× 7 with a
stride of 4 and dilation rates of S1 = [1, 2, 3, 4]. In the following two RCs, the convolution kernel size is 3× 3 with a stride of 2 and dilation rates of S2 = [1, 2, 3] and S3 = [1, 2], respectively. Since the spatial dimension of tokens decreases, there is no need to use large kernels and dilation rates. PCM in both RCs and NCs comprises three convolutional layers with a kernel size of 3× 3.

heading: 4 Experiments

heading: 4.1 Implementation details

text: We train and test the proposed ViTAE model on the standard ImageNet [38] dataset, which contains about 1.3 million images and covers 1k classes. Unless explicitly stated, the image size during training is set to 224× 224. We use the AdamW [48] optimizer with the cosine learning rate scheduler and uses the data augmentation strategy exactly the same as T2T [93] for a fair comparison, regarding the training strategies and the size of models. We use a batch size of 512 for training all our models and set the initial learning rate to be 5e-4. The results of our models can be found in Table 2, where all the models are trained for 300 epochs on 8 V100 GPUs. The models are built on PyTorch [57] and TIMM [82].

heading: 4.2 Comparison with the state-of-the-art

text: We compare our ViTAE with both CNN models and vision transformers with similar model sizes in Table 2. Both Top-1/5 accuracy and real Top-1 accuracy on the ImageNet validation set are reported. We categorize the methods into CNN models, vision transformers with learned IB, and vision transformers with introduced intrinsic IB. Compared with CNN models, our ViTAE-T achieves a 75.3% Top-1 accuracy, which is better than ResNet-18 with more parameters. The real Top-1 accuracy of the ViTAE model is 82.9%, which is comparable to ResNet-50 that has four more times of parameters than ours. Similarly, our ViTAE-S achieves 82.0% Top-1 accuracy with half of the parameters of ResNet-101 and ResNet-152, showing the superiority of learning both local and longrange features from specific structures with corresponding intrinsic IBs by design. Similar phenomena can also be observed when comparing ViTAE-T with MobileNetV1 [31] and MobileNetV2 [65], where ViTAE obtains better performance with fewer parameters. When compared with larger models which are searched according to NAS [73], our ViTAE-S achieves a similar performance when using 384× 384 images as input, which further shows the potential of vision transformers with intrinsic IB. In addition, among the transformers with learned IB, ViT is the first pure transformer model for visual recognition. DeiT shares the same structure with ViT but uses different data augmentation and training strategies to facilitate the learning of transformers. DeiT⚗denotes using an off-the-shelf CNN model as the teacher model to train DeiT, which introduces the intrinsic IB from CNN to transformer implicitly in a knowledge distillation manner, showing better performance than the vanilla ViT on the ImageNet dataset. It is exciting to see that our ViTAE-T with fewer parameters even outperforms the distilled model DeiT⚗, demonstrating the efficacy of introducing intrinsic IBs in transformers by design. Besides, compared with other transformers with explicit intrinsic IB, our ViTAE with fewer parameters also achieves comparable or better performance. For instance, ViTAE-T achieves comparable performance with LocalVit-T but has 1M fewer parameters, demonstrating the superiority of the proposed RCs and NCs in introducing intrinsic IBs.

heading: 4.3 Ablation study

text:
We use T2T-ViT [93] as our baseline model in the following ablation study of our ViTAE. As shown in Table 3, we investigate the hyper-parameter settings in RCs and NCs by isolating them separately. All the models are trained for 100 epochs on ImageNet and follow the same training setting and data augmentation strategy as described in Section 4.1.
We use X and× to denote whether or not the corresponding module is enabled during the experiments. If all columns under the RC and NC are marked × as shown in the first row, the model becomes the standard T2T-ViT model. “Pre” indicates the output features of PCM and MHSA are fused before FFN while “Post” indicates a late fusion strategy correspondingly. “BN” indicates whether PCM uses BN after the convolutional layer or not. “×3” in the first column denotes that the dilation rate set is the same in the three RCs. “[1, 2, 3, 4] ↓” denotes using lower dilation rates in deeper RCs, i.e., S1 = [1, 2, 3, 4], S2 = [1, 2, 3], S3 = [1, 2].
As can be seen, using a pre-fusion strategy and BN achieves the best 69.9% Top-1 accuracy among other settings. It is noteworthy that all the variants of NC outperform the vanilla T2T-ViT, implying the effectiveness of PCM, which introduces the intrinsic locality IB in transformers. It can also be observed that BN plays an important role in improving the model’s performance as it can help to alleviate the scale deviation between convolution’s and attention’s features. For the RC, we first investigate the impact of using different dilation rates in the PRM, as shown in the first column. As can be seen, using larger dilation rates (e.g., 4 or 5) does not deliver better performance. We suspect that larger dilation rates may lead to plain features in the deeper RCs due to the smaller resolution of feature maps. To
validate the hypothesis, we use smaller dilation rates in deeper RCs as denoted by [1, 2, 3, 4] ↓. As can be seen, it achieves comparable performance as [1, 2, 3]×. However, compared with [1, 2, 3, 4] ↓, [1, 2, 3]× increases the amount of parameters from 4.35M to 4.6M. Therefore, we select [1, 2, 3, 4] ↓ as the default setting. In addition, after using PCM in the RC, it introduces the intrinsic locality IB, and the performance increases to 71.7% Top-1 accuracy. Finally, the combination of RCs and NCs achieves the best accuracy at 72.6%, demonstrating the complementarity between our RCs and NCs.

heading: 4.4 Data efficiency and training efficiency

text:
To validate the effectiveness of the introduced intrinsic IBs in improving data efficiency and training efficiency, we compare our ViTAE with T2T-ViT at different training settings: (a) training them using 20%, 60%, and 100% ImageNet training set for equivalent 100 epochs on the full ImageNet training set, e.g., we employ 5 times epochs when using 20% data for training compared with using 100% data; and (b) training them using the full ImageNet training set for 100, 200, and 300 epochs respectively. The results are shown in Figure 1. As can be seen, ViTAE consistently outperforms the T2T-ViT baseline by a large margin in terms of both data efficiency and training efficiency. For example, ViTAE using only 20% training data achieves comparable performance with T2T-ViT using all data. When 60% training data are used, ViTAE significantly outperforms T2T-ViT using all data by about an absolute 3% accuracy. It is also noteworthy that ViTAE trained for only 100 epochs has outperformed T2T-ViT trained for 300 epochs. After training ViTAE for 300 epochs, its performance is significantly boosted to 75.3% Top-1 accuracy. With the proposed RCs and NCs, the transformer layers in our ViTAE only need to focus on modeling long-range dependencies, leaving the locality and multi-scale context modeling to its convolution counterparts, i.e., PCM and PRM. Such a “divide-and-conquer” strategy facilitates the training of vision transformers, making it possible to learn more efficiently with less training data and fewer training epochs.
To further validate the data efficiency of ViTAE model, we train the ViTAE model from scratch on the smaller datasets, i.e., Cifar10 and Cifar100. The results are summarized in Table 4. It can be viewed that with only 1/7 number of epochs, the ViTAE-T model achieves better classification
performance on Cifar10 dataset, with far fewer parameters (4.8M v.s. 86M), which further confirms ViTAE model’s data efficiency.

heading: 4.5 Generalization on downstream tasks

text: We further investigate the generalization of the proposed ViTAE models on downstream tasks by finetuning them on the training sets of several fine-grained classification tasks3, including Flowers [53], Cars [36], Pets [56], and iNaturalist19. We also fine-tune the proposed ViTAE models on Cifar10 [37] and Cifar100 [37]. The results are shown in Table 5. It can be seen that ViTAE achieves SOTA performance on most of the datasets using comparable or fewer parameters. These results demonstrate that the good generalization ability of our ViTAE.

heading: 4.6 Visual inspection of ViTAE

text:
To further analyze the property of our ViTAE, we first calculate the average attention distance of each layer in ViTAE-T and the baseline T2T-ViT-7 on the ImageNet test set, respectively. The results are shown in Figure 3. It can be observed that with the usage of PCM, which focuses on modeling locality, the transformer layers in the proposed NCs can better focus on modeling long-range dependencies, especially in shallow layers. In the deep layers, the average attention distances of ViTAE-T and T2T-ViT-7 are almost the same since modeling long-range dependencies is much more important.
These results confirm the effectiveness of the adopted “divide-and-conquer” idea in the proposed ViTAE, i.e., introducing the intrinsic locality IB from convolutions into vision transformers makes it possible that transformer layers only need to be responsible to long-range dependencies, since locality can be well modeled by convolutions in PCM.
Besides, we apply Grad-CAM [66] on the MHSA’s output in the last NC to qualitatively inspect ViTAE. The visualization results are provided in Figure 4. Compared with the baseline T2T-ViT, our ViTAE covers the single or multiple targets in the images
more precisely and attends less to the background. Moreover, ViTAE can better handle the scale variance issue as shown in Figure 4(b). Namely, it can precisely cover the birds no matter they are in small, middle, or large size. Such observations demonstrate that introducing the intrinsic IBs of locality and scale-invariance from convolutions to transformers helps ViTAE learn more discriminate features than the pure transformers.
3The performance of ViTAE on dense prediction tasks such as detection, segmentation, pose estimation, can be found in the supplementary material.
(a) (b)
Input
T2T-ViT
ViTAE
Figure 4: Visual inspection of T2T-ViT and ViTAE using Grad-CAM [66]. (a) Images containing multiple or single objects and the heatmaps. (b) Images containing the same class of objects at different scales and the heatmaps (Best viewed in color).

heading: 5 Limitation and discussion

text: In this paper, we explore two types of IBs and incorporate them into transformers through the proposed reduction and normal cells. With the collaboration of these two cells, our ViTAE model achieves impressive performance on the ImageNet with fast convergence and high data efficiency. Nevertheless, due to computational resource constraints, we have not scaled the ViTAE model and train it on largesize dataset, e.g., ImageNet-21K [38] and JFT-300M [30]. Although it remains unclear by now, we are optimistic about its scale property from the following preliminary evidence. As illustrated in Figure 2, our ViTAE model can be viewed as an intra-cell ensemble of complementary transformer layers and convolution layers owing to the skip connection and parallel structure. According to the attention distance analysis shown in Figure 3, the ensemble nature enables the transformer layers and convolution layers to focus on what they are good at, i.e., modeling long-range dependencies and locality. Therefore, ViTAE is very likely to learn better feature representation from large-scale data. Besides, we only study two typical IBs in this paper. More kinds of IBs such as constituting viewpoint invariance [64] can be explored in the future study.

heading: 6 Conclusion

text:
In this paper, we re-design the transformer block by proposing two basic cells (reduction cells and normal cells) to incorporate two types of intrinsic inductive bias (IB) into transformers, i.e., locality and scale-invariance, resulting in a simple yet effective vision transformer architecture named ViTAE. Extensive experiments show that ViTAE outperforms representative vision transformers in various respects including classification accuracy, data efficiency, training efficiency, and generalization ability on downstream tasks. We plan to scale ViTAE to the large or huge model size and train it on large-size datasets in the future study. In addition, other kinds of IBs will also be investigated. We hope that this study will provide valuable insights to the following studies of introducing intrinsic IB into vision transformers and understanding the impact of intrinsic and learned IBs.
Acknowledgement Dr. Jing Zhang is supported by the ARC project FL-170100117.

title: ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias


summaries: This paper introduces a novel adversarial fine-tuning method, which they apply to language model fine-tuning. Their method, named RIFT, takes an information-theoretic perspective to define the new loss; this paper is very clearly written and includes mostly useful figures that help explain their novel approach. They show strong experimental results with a number of baselines, and run an ablation study for the hyperparameters introduced by their approach.

summaries: This paper argues that applying adversarial training into the fine-tuning of the pre-trained language model suffers severely from catastrophic forgetting. It then proposes Robust Informative Fine-Tuning (RIFT) to encourage an objective model to retain the features learned from the pre-trained model. The results on sentiment analysis and natural language inference tasks show that the proposed method is able to outperform the strong baseline models in terms of Genetic and PWWS attacks.

summaries: This paper studies the catastrophic forgetting problem when fine-tuning pre-trained language models towards adversarial robustness. They argue that existing robust training methods, such as adversarial training, make the objective model deviated too much from the pre-trained model, resulting in bad generalization. Starting from an information-theoretical perspective, the authors propose RIFT, a fine-tuning method that maximizes the mutual information between the outputs of the objective model and the outputs of the pre-trained model when conditioned on the labels. The experimental results show that RIFT has a better performance than other baselines, which supports the better generalization as the authors claim. In addition, they give explanations why considering I(S;T|Y) in the objective is better than considering I(S;T).   ==== After Rebuttal ====  Thanks for the response. All my questions are properly resolved.

summaries: This paper proposes a new method for finetuning pretrained NLP models to achieve both high standard and high robust accuracy. They argue that finetuning on the more difficult and plentiful adversarial examples can lead to catastrophic forgetting, more so than standard finetuning of models. They propose an additional loss wherein the model is encouraged to get high accuracy while also retaining features from the pretrained weights. This improves the accuracy over standard finetuning and various other ways of regularizing adversarial finetuning.

summaries: The paper studies the problem of adversarial fine-tuning of pretrained NLP models, that is fine-tuning them in a way that makes them robust to adversarial attacks. It proposes a new method called RIFT. RIFT alters the usual adversarial training by adding a regularization term that maximizes the conditional mutual information between feature vectors produced by the original pretrained model and the currently fine-tuned one. The motivation is to alleviate catastrophic forgetting. The experiments show RIFT getting a slightly higher adversarial robustness than the baselines methods, such as e.g. using the L2 penalty instead of the proposed regularizer.

abstractText: The fine-tuning of pre-trained language models has a great success in many NLP fields. Yet, it is strikingly vulnerable to adversarial examples, e.g., word substitution attacks using only synonyms can easily fool a BERT-based sentiment analysis model. In this paper, we demonstrate that adversarial training, the prevalent defense technique, does not directly fit a conventional fine-tuning scenario, because it suffers severely from catastrophic forgetting: failing to retain the generic and robust linguistic features that have already been captured by the pre-trained model. In this light, we propose Robust Informative Fine-Tuning (RIFT), a novel adversarial fine-tuning method from an information-theoretical perspective. In particular, RIFT encourages an objective model to retain the features learned from the pre-trained model throughout the entire fine-tuning process, whereas a conventional one only uses the pre-trained weights for initialization. Experimental results show that RIFT consistently outperforms the state-of-the-arts on two popular NLP tasks: sentiment analysis and natural language inference, under different attacks across various pre-trained language models. 1.

authors:
Rahaf Aljundi
Min Lin
Baptiste Goujaud
Yoshua Bengio

title: Gradient based sample selection for online continual learning

venue: In NeurIPS,

authors:
Moustafa Alzantot
Yash Sharma
Ahmed Elgohary
Bo-Jhang Ho
Mani Srivastava
Kai-Wei Chang

title: Generating natural language adversarial examples

venue: In EMNLP,

authors:
Anish Athalye
Nicholas Carlini
David Wagner

title: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples

authors:
Yonatan Belinkov
Yonatan Bisk

title: Synthetic and natural noise both break neural machine translation

venue: In ICLR,

authors:
Ari S Benjamin
David Rolnick
Konrad Kording

title: Measuring and regularizing networks in function space

venue: In ICML,

authors:
Samuel R. Bowman
Gabor Angeli
Christopher Potts
Christopher D. Manning

title: A large annotated corpus for learning natural language inference

authors:
Nicholas Carlini
David Wagner

title: Towards evaluating the robustness of neural networks

venue: In SP,

authors:
Ciprian Chelba
Alex Acero

title: Adaptation of maximum entropy capitalizer: Little data can help a lot

venue: Computer Speech & Language,

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey Hinton

title: A simple framework for contrastive learning of visual representations

title: Frustratingly easy domain adaptation

venue: arXiv preprint arXiv:0907.1815,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: In NAACL,

authors:
Xinshuai Dong
Anh Tuan Luu
Rongrong Ji
Hong Liu

title: Towards robustness against natural language word substitutions

authors:
Javid Ebrahimi
Anyi Rao
Daniel Lowd
Dejing Dou

title: Hotflip: White-box adversarial examples for text classification

venue: In ACL,

authors:
Sayna Ebrahimi
Mohamed Elhoseiny
Trevor Darrell
Marcus Rohrbach

title: Uncertaintyguided continual learning with bayesian neural networks

authors:
Steffen Eger
Gözde Gül Şahin
Andreas Rücklé
Ji-Ung Lee
Claudia Schulz
Mohsen Mesgar
Krishnkant Swarnkar
Edwin Simpson
Iryna Gurevych

title: Text processing like humans do: Visually attacking and shielding nlp systems

authors:
Ji Gao
Jack Lanchantin
Mary Lou Soffa
Yanjun Qi

title: Black-box generation of adversarial text sequences to evade deep learning classifiers

venue: In SPW

authors:
Ian J Goodfellow
Mehdi Mirza
Da Xiao
Aaron Courville
Yoshua Bengio

title: An empirical investigation of catastrophic forgetting in gradient-based neural networks

venue: arXiv preprint arXiv:1312.6211,

authors:
Ian J Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

authors:
Michael Gutmann
Aapo Hyvärinen

title: Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

venue: In AISTATS,

authors:
Kaiming He
Haoqi Fan
Yuxin Wu
Saining Xie
Ross Girshick

title: Momentum contrast for unsupervised visual representation learning

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Geoffrey Hinton
Oriol Vinyals
Jeff Dean

title: Distilling the knowledge in a neural network

venue: arXiv preprint arXiv:1503.02531,

authors:
R Devon Hjelm
Alex Fedorov
Samuel Lavoie-Marchildon
Karan Grewal
Phil Bachman
Adam Trischler
Yoshua Bengio

title: Learning deep representations by mutual information estimation and maximization

authors:
Hossein Hosseini
Sreeram Kannan
Baosen Zhang
Radha Poovendran

title: Deceiving google’s perspective api built for detecting toxic comments

venue: arXiv preprint arXiv:1702.08138,

authors:
Jeremy Howard
Sebastian Ruder

title: Universal language model fine-tuning for text classification

authors:
Xinting Hu
Kaihua Tang
Chunyan Miao
Xian-Sheng Hua
Hanwang Zhang

title: Distilling causal effect of data in class-incremental learning

authors:
Po-Sen Huang
Robert Stanforth
Johannes Welbl
Chris Dyer
Dani Yogatama
Sven Gowal
Krishnamurthy Dvijotham
Pushmeet Kohli

title: Achieving verified robustness to symbol substitutions via interval bound propagation

authors:
Andrew Ilyas
Shibani Santurkar
Dimitris Tsipras
Logan Engstrom
Brandon Tran
Aleksander Madry

title: Adversarial examples are not bugs, they are features

authors:
Mohit Iyyer
John Wieting
Kevin Gimpel
Luke Zettlemoyer

title: Adversarial example generation with syntactically controlled paraphrase networks

authors:
Robin Jia
Percy Liang

title: Adversarial examples for evaluating reading comprehension systems

authors:
Robin Jia
Aditi Raghunathan
Kerem Göksel
Percy Liang

title: Certified robustness to adversarial word substitutions

authors:
Di Jin
Zhijing Jin
Joey Tianyi Zhou
Peter Szolovits

title: Is bert really robust? natural language attack on text classification and entailment

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
James Kirkpatrick
Razvan Pascanu
Neil Rabinowitz
Joel Veness
Guillaume Desjardins
Andrei A Rusu
Kieran Milan
John Quan
Tiago Ramalho
Agnieszka Grabska-Barwinska

title: Overcoming catastrophic forgetting in neural networks

venue: Proceedings of the national academy of sciences,

authors:
Alexey Kurakin
Ian Goodfellow
Samy Bengio

title: Adversarial examples in the physical world, 2016

title: Adversarial machine learning at scale

authors:
Cheolhyoung Lee
Kyunghyun Cho
Wanmo Kang

title: Mixout: Effective regularization to finetune large-scale pretrained language models

authors:
Yingzhen Li
John Bradshaw
Yash Sharma

title: Are generative classifiers more robust to adversarial attacks

authors:
Zhizhong Li
Derek Hoiem

title: Learning without forgetting

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Bin Liang
Hongcheng Li
Miaoqiang Su
Pan Bian
Xirong Li
Wenchang Shi

title: Deep text classification can be fooled

authors:
Yinhan Liu
Myle Ott
Naman Goyal
Jingfei Du
Mandar Joshi
Danqi Chen
Omer Levy
Mike Lewis
Luke Zettlemoyer
Veselin Stoyanov

title: Roberta: A robustly optimized bert pretraining approach

authors:
David Lopez-Paz
Marc’Aurelio Ranzato

title: Gradient episodic memory for continual learning

authors:
Ilya Loshchilov
Frank Hutter

title: Decoupled weight decay regularization

authors:
Andrew L. Maas
Raymond E. Daly
Peter T. Pham
Dan Huang
Andrew Y. Ng
Christopher Potts

title: Learning word vectors for sentiment analysis

authors:
Aleksander Madry
Aleksandar Makelov
Ludwig Schmidt
Dimitris Tsipras
Adrian Vladu

title: Towards deep learning models resistant to adversarial attacks

authors:
Michael McCloskey
Neal J Cohen

title: Catastrophic interference in connectionist networks: The sequential learning problem

venue: In Psychology of learning and motivation. Elsevier,

authors:
Tomas Mikolov
Ilya Sutskever
Kai Chen
Greg Corrado
Jeffrey Dean

title: Distributed representations of words and phrases and their compositionality

venue: NeurIPS,

authors:
Takeru Miyato
Andrew M Dai
Ian Goodfellow

title: Adversarial training methods for semisupervised text classification

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Representation learning with contrastive predictive coding

venue: arXiv preprint arXiv:1807.03748,

authors:
Nicolas Papernot
Patrick McDaniel
Somesh Jha
Matt Fredrikson
Z Berkay Celik
Ananthram Swami

title: The limitations of deep learning in adversarial settings

venue: In EuroS&P

authors:
Jeffrey Pennington
Richard Socher
Christopher D Manning

title: Glove: Global vectors for word representation

authors:
Matthew E Peters
Mark Neumann
Mohit Iyyer
Matt Gardner
Christopher Clark
Kenton Lee
Luke Zettlemoyer

title: Deep contextualized word representations

authors:
Ben Poole
Sherjil Ozair
Aaron Van Den Oord
Alex Alemi
George Tucker

title: On variational bounds of mutual information

authors:
Danish Pruthi
Bhuwan Dhingra
Zachary C Lipton

title: Combating adversarial misspellings with robust word recognition

authors:
Alec Radford
Karthik Narasimhan
Tim Salimans
Ilya Sutskever

title: Improving language understanding by generative pre-training

authors:
Sylvestre-Alvise Rebuffi
Alexander Kolesnikov
Georg Sperl
Christoph H Lampert

title: icarl: Incremental classifier and representation learning

authors:
Emily Reif
Ann Yuan
Martin Wattenberg
Fernanda B Viegas
Andy Coenen
Adam Pearce
Been Kim

title: Visualizing and measuring the geometry of bert

authors:
Shuhuai Ren
Yihe Deng
Kun He
Wanxiang Che

title: Generating natural language adversarial examples through probability weighted word saliency

authors:
Marco Tulio Ribeiro
Sameer Singh
Carlos Guestrin

title: Semantically equivalent adversarial rules for debugging nlp models

authors:
Leslie Rice
Eric Wong
Zico Kolter

title: Overfitting in adversarially robust deep learning

title: Improved deep metric learning with multi-class n-pair loss objective

authors:
Nitish Srivastava
Geoffrey Hinton
Alex Krizhevsky
Ilya Sutskever
Ruslan Salakhutdinov

title: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning

authors:
David Stutz
Matthias Hein
Bernt Schiele

title: Disentangling adversarial robustness and generalization

venue: In CVPR,

authors:
Christian Szegedy
Wojciech Zaremba
Ilya Sutskever
Joan Bruna
Dumitru Erhan
Ian Goodfellow
Rob Fergus

title: Intriguing properties of neural networks

authors:
Ian Tenney
Dipanjan Das
Ellie Pavlick. Bert

title: re-discovers the classical nlp pipeline

authors:
Yonglong Tian
Dilip Krishnan
Phillip Isola

title: Contrastive representation distillation

authors:
Yonglong Tian
Chen Sun
Ben Poole
Dilip Krishnan
Cordelia Schmid
Phillip Isola

title: What makes for good views for contrastive learning

authors:
Florian Tramer
Nicholas Carlini
Wieland Brendel
Aleksander Madry

title: On adaptive attacks to adversarial example defenses

authors:
Michael Tschannen
Josip Djolonga
Paul K Rubenstein
Sylvain Gelly
Mario Lucic

title: On mutual information maximization for representation learning

authors:
Dimitris Tsipras
Shibani Santurkar
Logan Engstrom
Alexander Turner
Aleksander Madry

title: Robustness may be at odds with accuracy

venue: arXiv preprint arXiv:1805.12152,

authors:
Laurens Van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

venue: Journal of machine learning research,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Li Wan
Matthew Zeiler
Sixin Zhang
Yann Le Cun
Rob Fergus

title: Regularization of neural networks using dropconnect

authors:
Tongzhou Wang
Phillip Isola

title: Understanding contrastive representation learning through alignment and uniformity on the hypersphere

authors:
Georg Wiese
Dirk Weissenborn
Mariana Neves

title: Neural domain adaptation for biomedical question answering

venue: In CoNLL,

authors:
Eric Wong
Frank R Schmidt
J Zico Kolter

title: Wasserstein adversarial examples via projected sinkhorn iterations

authors:
Zhilin Yang
Zihang Dai
Yiming Yang
Jaime Carbonell
Ruslan Salakhutdinov
Quoc V Le

title: Xlnet: Generalized autoregressive pretraining for language understanding

authors:
Yuan Zang
Fanchao Qi
Chenghao Yang
Zhiyuan Liu
Meng Zhang
Qun Liu
Maosong Sun

title: Word-level textual adversarial attacking as combinatorial optimization

authors:
Hongyang Zhang
Yaodong Yu
Jiantao Jiao
Eric P. Xing
Laurent El Ghaoui
Michael I. Jordan

title: Theoretically principled trade-off between robustness and accuracy

authors:
Huangzhao Zhang
Hao Zhou
Ning Miao
Lei Li

title: Generating fluent adversarial examples for natural languages

authors:
Tianyi Zhang
Felix Wu
Arzoo Katiyar
Kilian Q Weinberger
Yoav Artzi

title: Revisiting few-sample bert fine-tuning

authors:
Zhengli Zhao
Dheeru Dua
Sameer Singh

title: Generating natural adversarial examples

heading: 1 Introduction

text:
Deep models are well-known to be vulnerable to adversarial examples [64, 19, 50, 35]. For instance, fine-tuned models pre-trained on very large corpora can be easily fooled by word substitution attacks using only synonyms [2, 58, 32, 12]. This has raised grand security challenges to modern NLP systems, such as spam filtering and malware detection, where pre-trained language models like BERT [11] are widely deployed.
Attack algorithms [19, 7, 76, 40, 2] aim to maliciously generate adversarial examples to fool a victim model, while adversarial defense aims at building robustness against them. Among the defense methods, adversarial training [64, 19, 48, 45] is the most effective one [3]. It updates model parameters using perturbed adversarial samples generated on-the-fly and yields consistently robust performance even against the challenging adaptive attacks [3, 68].
However, despite its effectiveness in training from scratch, adversarial training may not directly fit the current NLP paradigm, the fine-tuning of pre-trained language models. First, fine-tuning per se suffers from catastrophic forgetting [46, 18, 34], i.e., the resultant model tends to over-fit to a small fine-tuning data set, which may deviate too far from the pre-trained knowledge [25, 81]. Second, adversarially fine-tuned models are more likely to forget: adversarial samples are usually
1Our code will be available at https://github.com/dongxinshuai/RIFT-NeurIPS2021.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
out-of-distribution [38, 63], thus they are generally inconsistent with the pre-trained model. As a result, adversarial fine-tuning fails to memorize all the robust and generic linguistic features already learned during pre-training [65, 57], which are however very beneficial for a robust objective model.
Addressing forgetting is essential for achieving a more robust objective model. Conventional wisdom such as pre-trained weight decay [8, 10] and random mixout [37] mitigates forgetting by constraining lp distance between the two models’ parameters. Though effective to some extent, however, it is limited because change in the model parameter space only serves as an imperfect proxy for that in the function space [5]. Besides, the extent to which an encoder fails to retain information also depends on the input distribution. Therefore, a better way to encourage memorization is favorable.
In this paper, we follow an information-theoretical lens to look into the forgetting problem: we employ mutual information to measure how well an objective model memorizes the useful features captured before. This motivates our novel adversarial fine-tuning method, Robust Informative FineTuning (RIFT). In addition to fitting a down-stream task as conventions, RIFT maximizes the mutual information between the output of an objective model and that of the pre-trained model conditioned on the class label. It encourages an objective model to continuously retain useful information from the pre-trained one throughout the whole fine-tuning process, whereas the conventional one only makes use of the pre-trained weights for initialization. We illustrate the overall objective of RIFT in Figure 1 and summarize the major contributions of this paper as follows:
• To the best of our knowledge, we are the first to investigate the intensified phenomenon of catastrophic forgetting in the adversarial fine-tuning of pre-trained language models.
• To address the forgetting problem and achieve more robust models, we propose a novel approach named Robust Informative Fine-Tuning (RIFT) from an information-theoretical perspective. RIFT enables an objective model to retain robust and generic linguistic features throughout the whole fine-tuning process and thus enhance robustness against adversarial examples.
• We empirically evaluate RIFT on two prevailing NLP tasks, sentiment analysis and natural language inference, where RIFT consistently outperforms the state-of-the-arts in terms of robustness, under different attacks across various pre-trained language models.

heading: 2 Methodology

heading: 2.1 Notations and Problem Setting

text:
In this paper, we focus on the text classification task to introduce our method, while it can be easily extended to other NLP tasks. We suppose random variables X,Y ∼ pD(x, y), where X represents the textual input, Y represents the class label, pD is the data distribution, and x, y are the observed values. Our goal is to build a classifier q(y∣Fs(x)), where Fs(⋅), referred to as our objective model
(a) In adversarial fine-tuning, the relative L2 distance continuously grows as the fine-tuning proceeds. (b) Trading clean accuracy off against robustness (by increasing β in Eq. 2) induces increased relative L2 distance at the last epoch.
Figure 2: Relative L2 distance between the pre-trained model and the objective model in the parameter space, under different fine-tuning schemes on IMDB, with small learning rate 2e-5.
in the rest of this paper, is a feature extractor fine-tuned from the encoder of a pre-trained language model, e.g., Transformer [72], and q(⋅∣⋅) is parameterized by using an MLP with softmaxed output. We favor a classifier that is robust against adversarial attacks [64, 2], i.e., maintaining high accuracy even given adversarial examples as inputs.

heading: 2.2 Adversarial Fine-Tuning Suffers From Forgetting

text:
Adversarial training [64, 19, 45] is currently the most effective defense technique [3]. Its training objective can be generally formulated as:
min[ E x,y∼pD [ max x̂∈B(x) L(x, x̂, y)]], (1)
where B(x) defines the allowed perturbation space around x, and L is a loss to encourage correct predictions both given vanilla samples and given perturbed examples. Such an objective function can also be applied to the fine-tuning of a pre-trained language model towards robustness, and we refer to it as adversarial fine-tuning. The loss function in Eq. 1 can be specified following Miyato et al. [48], Zhang et al. [79] in a semi-supervised fashion as:
− log q(y∣Fs(x)) + βKL(q(⋅∣Fs(x))∣∣q(⋅∣Fs(x̂))), (2) where the Kullback–Leibler divergence encourages invariant predictions between x and x̂.
Despite its effectiveness in training from scratch, such an adversarial objective function may not directly fit the fine-tuning scenario. To begin with, fine-tuning itself suffers from catastrophic forgetting [46, 18, 34]. During fine-tuning, an objective model Fs(⋅) tends to continuously deviate away from the pre-trained one to fit a down-stream task [25, 81], and thus the useful information already captured before is less utilized as the fine-tuning proceeds. Further, adversarial fine-tuning suffers even more from the forgetting problem, the reasons of which are given as what follows.
(i) Adversarial Fine-Tuning Tends to Forget: Adversarial fine-tuning targets at tackling adversarial examples, which are generally out of the manifold [38, 63] of the pre-training corpora. To additionally handle them, an objective model would be fine-tuned towards a solution that is far away from the optimization starting point, i.e., the pre-trained model. Figure 2 (b) empirically shows this point: by increasing β in Eq. 2, we emphasize more on robustness instead of vanilla accuracy, and consequently, at the last epoch the distance between models also increases. Besides, adversarial fine-tuning often entails more iterations to converge (several times of normal fine-tuning), which further intensifies the forgetting problem, as the objective model is continuously deviating away as shown in Figure 2 (a).
(ii) Adversarial Fine-Tuning Needs to Memorize: Overfitting to training data is a dominant phenomenon throughout the adversarial training/fine-tuning process [60]. To mitigate overfitting and generalize better, adversarial fine-tuning should retain all the generalizable features already captured by the pre-trained model [11, 41]. In addition, adversarial fine-tuning favors an objective model that
extracts features invariant to perturbations [70, 28] for robustness. As such, all those generalizable and robust linguistic information captured during pre-training [11, 65] are particularly beneficial and should be memorized. We empirically validates that encouraging memorization does improve both generalization and robustness in Sec. 3.5.
To address forgetting, previous methods such as pre-trained weight decay [8, 10] and random mixout [37] have shown their effectiveness in stabilizing fine-tuning [81]. However, they focus only on the parameter space, in which they encourage an object model to be similar to the pre-trained one. Distance in the parameter space can only approximately characterize the change in the function space, and fails to take the data distribution into consideration. A more natural way to capture the extent to which a model memorizes or forgets, should be using the mutual information between outputs of the two models, and we provide our solution as follows.

heading: 2.3 Informative Fine-Tuning

text:
We first use an information-theoretical perspective to look into how a pre-trained model should be leveraged throughout the whole fine-tuning process.
We define random variable T = Ft(X) as the feature of X extracted by the pre-trained language model Ft(⋅), and t as the observed value of T . Similarly, we define random variable S = Fs(X) as the feature of X extracted by our objective model Fs(⋅), and s as the observed value of S. We formulate an overall fine-tuning objective as follows. The motivation is to train Fs(⋅) such that S is capable of predicting Y , as well as preserving the information from T .
max I(S; Y,T ), (3)
i.e., maximizing the mutual information between (i) the feature extracted by the objective model and (ii) the class label plus the feature extracted by the pre-trained model. Since the pre-trained model Ft(⋅) is a fixed deterministic function, the fine-tuning objective in Eq. 3 in essense encourages Fs(⋅) to output features that contain as much information as possible for predicting Y and T . This enables the objective model to learn from the pre-trained language model via T throughout the whole fine-tuning process, and thus helps address the forgetting problem.
However, the objective defined in Eq. 3 is generally hard to optimize directly. Therefore, we decompose Eq. 3 into two terms as follows:
I(S; Y,T ) = I(S;Y ) + I(S;T ∣ Y ), (4)
where I(S;Y ) measures how well the output of our objective model can predict the label, and I(S;T ∣ Y ) measures when conditioned on the class label, how well the output features of the two models can predict each other. Visualization of each component can be seen in Figure 1 for more intuitive understandings. We next introduce how each term in the right-hand side of Eq. 4 can be transformed into a tractable lower bound for optimization.
(i) Maximizing I(S;Y): We treat q(y∣s), which is the classification layer that takes the features extracted by the objective model as input, as a variational distribution of p(y∣s), and derive a variational lower bound on I(S;Y ) following variational inference [33] as follows:
I(S;Y ) =H(Y ) −Ex,y∼pD [− log q(y∣s)] +KL(p(⋅∣s)∣∣q(⋅∣s)) (5) ≥H(Y ) −Ex,y∼pD [− log q(y∣s)], (6)
whereH(Y ) is a constant measuring the shannon entropy of Y , and EX,Y [− log q(y∣s)] is essentially the cross-entropy loss using q(y∣s) for classification. Then, the objective of maximizing I(S;Y ) can be achieved by minimizing EX,Y [− log q(y∣s)] instead. (ii) Maximizing I(S;T| Y): The definition of the conditional mutual information I(S;T ∣ Y ) is as follows:
I(S;T ∣ Y ) = Ey∼pD(y)[I(S;T )∣ Y = y] = Ey∼pD(y)[ Ex∼pD(x∣y)[log p(s, t∣y)
p(s∣y)p(t∣y)]]. (7)
To achieve a tractable objective for maximizing Eq. 7, we employ noise contrastive estimation [20, 49] and derive a lower bound −Linfo on the conditional mutual information. This can be summarized in the following Lemma (the proof of which can be found in Appendix A.2):
Lemma 1. Given {xi, y}Ni=1 that is sampled i.i.d. from pD(x∣y), si = Fs(xi), and ti = Ft(xi), I(S;T ∣ Y ) is lower bounded by −Linfo = Ey∼pD(y)[E{xi,y}Ni=1[ 1 N ∑ N i=1 log efy(si,ti)
∑Nj=1 e fy(si,tj) + logN]], and fy is a score function indexed by y.
By leveraging Lemma 1, Linfo can be computed using a batch of samples and then minimized for maximizing I(S;T ∣ Y ). The score function fy is defined as the inner product after non-linear projections into a space of hyper-sphere following [9] as fy(a, b) = 1τ ⟨g1y(a),g 2 y(b)⟩
∥g1y(a)∥2∥g 2 y(b)∥2
, where g1y and
g2y are parameterized by using MLPs.

heading: 2.4 Conditional Mutual Information Better Fits A Down-Stream Task

text:
Above we have introduced the training objective of informative fine-tuning and decomposed it into I(S;Y ) and I(S;T ∣ Y ) for optimization. However, one may wonder why not maximize I(S;T ) directly instead of I(S;T ∣ Y ). From an information-theoretical perspective, if we maximize I(S;Y ) and I(S;T ), the intersection of the three circles in Figure 1, i.e., the interaction information of them, is repeatedly optimized and might induce confliction. To further elaborate this point, we look into contrastive loss and give an explanation as follows.
We consider contrastive learning by decomposing it into the encouragement of alignment and uniformity following [74]. For example, maximizing I(S;T ) =H(S) −H(S∣T ) is to decrease H(S∣T ) and increase H(S). Decreasing H(S∣T ) corresponds to encouraging alignment in contrastive learning, which aims to align si and ti in the sense of inner product when they are projected into a hyper-sphere. In the meanwhile, increasing H(S) corresponds to uniformity in contrastive learning, which encourages si and sj , where i ≠ j, to diffuse over the whole sub-space as uniformly as possible. It can be seen in Figure 3 (c) that, in the space of S1, alignment is achieved in that points from two different circles but with the same color are aligned, and uniformity is achieved in that all points diffuse over the whole S1. However, when maximizing I(S;T ), the uniformity in the contrastive loss is encouraged over the whole data manifold. Diffusing uniformly over the whole data manifold can be against the objective of maximizing I(S;Y ), which aims to separate the whole data manifold into class-specific parts as shown in Figure 3 (b). In contrast, maximizing I(S;T ∣ Y ) only encourages uniformity inside each class-specific data manifold, which is complementary to maximizing I(S;Y ), as show in Figure 3 (d). In the meanwhile, the alignment is still enforced. More empirical support can be found in Sec. 3.4.
Algorithm 1 RIFT Input: dataset D, hyper-parameters of AdamW [43] Output: the model parameters θ and φ
1: Initialize θ using the pre-trained model, and initialize φ and ϕ randomly. 2: repeat 3: Sample y ∼ pD(y) and then {xi, y}Ni=1 ∼ pD(x∣y) 4: for every xi, y in the mini-batch {xi, y}Ni=1 do 5: Find x̂i by solving Eq. 8; 6: end for 7: Compute the loss function defined in Eq. 11 and update θ, φ, and ϕ by gradients. 8: until the training converges.

heading: 2.5 Robust Informative Fine-Tuning

text:
In this section, we are going to put the objective function of informative fine-tuning into the adversarial context and introduce Robust Informative Fine-Tuning (RIFT).
To robustly train a model, the adversarial examples x̂ for training should be formulated and generated first. As our end goal is to enhance robustness in down-stream tasks, the generation process of adversarial examples should focus on preventing a model from predicting the ground truth label. However, using label for generating adversarial examples in training often induces label leaking [36], i.e., the generated adversarial examples contains label information which is used as a trivial short-cut by a classifier. As such, we follow [48, 79] to generate x̂ in a self-supervised fashion:
x̂ = argmax x′∈B(x) KL(q(⋅∣Fs(x))∣∣q(⋅∣Fs(x′))). (8)
By solving Eq. 8, x̂ is found to induce the most different prediction from that of a vanilla sample x in terms of KL, inside the attack space B(x). Next, we introduce how to robustly optimize each objective in the informative fine-tuning by using x̂.
(i) Robustly Maximizing I(S;Y): As shown in Sec. 2.3, maximizing I(S;Y ) can be achieved by minimizing a cross-entropy loss instead. To encourage adversarial robustness, this cross-entropy loss can be upgraded to encourage both correct predictions on and invariant predictions between x and x̂ [48, 79]. We formulate such an objective function as follows:
min θ,φ Lr-task, Lr-task = E x,y∼pD [ − log q(y∣Fs(x)) + βKL(q(⋅∣Fs(x))∣∣q(⋅∣Fs(x̂)))], (9)
where θ denotes the parameters of Fs(⋅) and φ denotes the parameters of q(⋅∣⋅). Minimizing Lr-task in Eq. 9 corresponds to adversarilly maximizing I(S;Y ) for robust performance in a down-stream task. By doing so, the adversarial example x̂ is generated by Eq. 8 first and then both x and x̂ are used to optimize the model parameters θ and φ.
(ii) Robustly Maximizing I(S;T| Y): We aim to maximize the conditional mutual information I(S;T ∣ Y ), but under an adversarial distribution of input data. We formulate such a term as I(Ŝ;T ∣ Y ), where random variable Ŝ = Fs(X̂), X̂ ∼ padv(x̂∣x, θ, φ,B), and sampling from padv, the adversarial distribution of input data, is to generate adversarial example x̂ by Eq. 8.
To optimize I(Ŝ;T ∣ Y ), we propose −Lr-info as a lower bound on it, and formulate the objective to minimize Lr-info as follows (similar to −Linfo by using Lemma 1):
min θ,ϕ Lr-info, Lr-info = E y∼pD(y) [ E {xi,y}Ni=1∼pD(x∣y)
[ 1 N
N ∑ i=1 − log e fy(ŝi,ti) ∑Nj=1 efy(ŝi,tj) − logN]], (10)
where ŝi = Fs(x̂i), ti = Ft(xi), and ϕ denotes the parameters of all the score functions fy. By Eq. 10, we are able to encourage Fs(⋅) to retain information from Ft(⋅) in a robust fasion. Noted that, in Lr-info we do not use x̂i to extract features from the pre-trained model. This follows the spirit of Knowledge Distillation [22] (though the setting is different in that a student is expected to perform identically to a teacher in knowledge distillation, while in fine-tuning it does not): the data used to extract features of a teacher should be inside the domain for which the teacher is trained. A
down-stream NLP task, e.g., sentiment analysis, has a data domain that is generally a sub-domain of the pre-training corpora, while x̂i can be significantly off the pre-training data manifold.
Objective Function of Robust Informative Fine-Tuning (RIFT): The objective function of RIFT is a combination of Lr-task and Lr-info, defined as follows:
min θ,φ,ϕ
Lr-task + αLr-info, (11)
where the hyper-parameter α controls to what extent we encourage an objective model to absorb information from the pre-trained one (ablation on α can be seen in Sec. 3.5). We summarize the whole training process in Algorithm 1 and include more implementation details in Appendix A.1.

heading: 3 Experiments

heading: 3.1 Experimental Setting

text:
Tasks and Datasets: We evaluate the robust accuracy and compare our method with the state-of-thearts on: (i) Sentiment analysis using the IMDB dataset [44]. (ii) Natural language inference using the SNLI dataset [6]. We mainly focus on robustness against adversarial word substitutions [2, 58], as such attacks preserve the syntactic and semantics very well [31, 78, 12], and are very hard to detect even by humans. Under this attack setting, any word in the input sequence can be substituted by a semantically similar word of it (often its synonym). We evaluate robustness on 1000 random examples from the testset of IMDB and SNLI respectively following [31, 12].
Model Architectures: We examine our methods and compare with state-of-the-arts on the following two prevailing pre-trained language models: (i) BERT-base-uncased [11]. (ii) RoBERTa-base [41].
Attack Algorithms: Two powerful attacks are employed: (i) Genetic [2] based on population algorithm. Aligned with [31, 12], the population size and iterations are set as 60 and 40 respectively. (ii) PWWS [58] based on word saliency. We only attack hypothesis on SNLI aligned with [31, 12].
Substitution Set: We follow [31, 12] to use the substitution set from [2], and the same language model constraint is applied to Genetic attacks and not to PWWS attacks.

heading: 3.2 Comparative Methods

text:
(i) Standard Fine-Tuning: The standard fine-tuning process first initializes the objective model by the pre-trained weight, and then use cross-entropy loss to fine-tune the whole model.
(ii) Adversarial Fine-Tuning Baseline (Adv-Base): We employ the state-of-the-art defense against word substitutions, ASCC-Defense [12], as the adversarial fine-tuning baseline. This method is not initially proposed for pre-trained language models but can readily extend to perform adversarial fine-tuning. During fine-tuning, adversarial example x̂, which is a sequence of convex combinations, is generated first and then both x and x̂ are used for optimization using objective defined in Eq. 9.
(iii) Adv + Pre-Trained Weight Decay (Adv-PTWD): Pre-trained weight decay [8, 10] penalizes λ∥Wobj −Wpre∥2, and mitigates catastrophic forgetting [75, 37]. We combine it with the adversarial baseline for comparisons and λ is chosen as 0.01 on IMDB and 0.005 on SNLI for best robustness.
(iv) Adv + Mixout (Adv-Mixout): Motivated by Dropout [62] and DropConnect [73], Mixout [37] is proposed to addresses catastrophic forgetting in fine-tuning. At each iteration each parameter is replaced by its pre-trained counter-part with probability m. We combine it with the adversarial fine-tuning baseline for comparisons with our method and m is chosen as 0.6 for best robustness.
(v) Robust Informative Fine-Tuning (RIFT): The proposed adversarial fine-tuning method. It can be deemed as the adversarial fine-tuning baseline plus the Lr-info term. We set τ as 0.2 for all score functions fy. For best robust accuracy, α is chosen as 0.1 and 0.7 on IMDB and SNLI respectively. Ablation study on α can be seen in Sec. 3.5.
For fair comparisons, all compared adversarial fine-tuning methods use the same β on a same dataset, i.e., β = 10 on IMDB and β = 5 on SNLI, both of which are chosen for the best robust accuracy. Early stopping [60] is used for all compared methods according to best robust accuracy More implementation details and runtime analysis can be found in Appendix A.1 and A.3.

heading: 3.3 Main Result

text:
In this section we compare our method with state-of-the-arts by robustness under attacks. As shown in Tables 1 and 2., RIFT consistently achieves the best robust performance among state-of-the-arts in all datasets across different pre-trained language models under all attacks. For instance, on IMDB our method outperforms the RoBERTa-based runner-up method by 2.9% under Genetic attacks and 1.9% under PWWS attacks. On SNLI based on BERT, we surpass the runner-up method by 1.2% under Genetic attacks and 1.1% under PWWS attacks. In addition, RIFT consistently improves robustness upon the adversarial fine-tuning baseline, while the improvements by other methods are not stable. We observe that on SNLI, we surpass the runner-up by a relatively small margin compared to IMDB. This may relate to the dataset property: input from SNLI has a smaller attack space (on average 6.54
combinations of substitutions on SNLI compared to 6108 on IMDB), and thus smaller absolute space left for improvement. For results of vanilla accuracy please refer to Appendix A.4.

heading: 3.4 Conditional Mutual Information Does Fit a Down-Stream Task Better

text: In this section we empirically validate that maximizing I(S;T ∣ Y ) cooperates with a down-stream task better than maximizing I(S;T ) does. We plot two sets of results of our method with maximizing I(S;T ∣ Y ) and I(S;T ) respectively (using similar noise contrastive loss with all other hyper-parameters the same). As shown in Table 3, RIFT with maximizing I(S;T ∣ Y ) consistently outperforms that with maximizing I(S;T ) in terms of robustness under all attacks on both IMDB and SNLI, which serves as an empirical validation of Sec. 2.4.

heading: 3.5 Ablation Study and Tradeoff Curve Between Robustness and Vanilla Accuracy

text:
In this section we conduct ablation study on α, which is the weight of Lr-info in Eq. 11. It aims to control to what extent the objective model absorbs information from the pre-trained one. As shown in Table 4, a good value of α improves both vanilla accuracy and robust accuracy; e.g., on IMDB increasing α from 0 to 0.1 results in increased vanilla accuracy by 0.2% and increased robust accuracy under Genetic attacks by 2.4%. This demonstrates that RIFT does motivate the objective model to retain robust and generalizable features that are beneficial to both robustness and vanilla accuracy. When α goes too large, the objective of fine-tuning would focus too much on preserving as much information from the pre-trained model as possible, and thus ignores the down-stream task.
One may wonder whether informative fine-tuning itself can improve the clean accuracy upon normal fine-tuning. The answer is yes; e.g., on IMDB using RoBERTa, informative fine-tuning improves about 0.3% clean accuracy upon normal fine-tuning baseline. The improvement is not very significant as normal fine-tuning targets at vanilla input only and thus suffers less from the forgetting problem. One common problem in contrastive loss is that the complexity grows quadratically with N , but larger N contributes to tighter bound on the mutual information [49, 53]. One potential improvement is to maintain a dictionary updated on-the-fly like [21] and we will leave it for future exploration.
We finally show the trade-off curve between robustness and vanilla accuracy in Figure 4. As shown, RIFT outperforms the state-of-the-arts in terms of both robust accuracy and vanilla accuracy. It again validates that RIFT indeed helps the objective model capture robust and generalizable information to improve both robustness and vanilla accuracy, rather than trivially trading off one against another.

heading: 4 Related Work

text:
Pre-Trained Language Models: Language modeling aims at estimating the probabilistic density of the textual data distribution. Classicial language modeling methods vary from CBOW, Skip-Gram [47], to GloVe [51] and ELMo [52]. Recently, Transformer [72] has demonstrated its power in language modeling, e.g., GPT [55] and BERT [11]. The representations learned by deep pre-trained language models are shown to be universal and highly generalizable [77] [41], and fine-tuning pre-trained language models becomes a paradigm in many NLP tasks [25, 11].
Adversarial Robustness: DNNs have achieved success in many fields, but they are susceptible to adversarial examples [64]. In NLP, attacks algorithms include char-level modifications [24, 13, 4, 17, 15, 54], sequence-level manipulations [29, 59, 30, 82], and adversarial word substitutions [2, 40, 58, 32, 80, 78]. For defenses, adversarial training [64, 19, 45] generates adversarial examples on-the-fly and is currently the most effective. Miyato et al. [48] first introduce adversarial training to
NLP using L2-ball to model perturbations, and later more geometry-aligned methods like first-order approximation[13], axis-aligned bound [31, 27], and convex hull [12], become favorable.
Fine-Tuning and Catastrophic Forgetting: The fine-tuning of pre-trained language models [25, 11] can be very unstable [11], as during fine-tuning the objective model can deviate too much from the pre-trained one, and easily over-fits to small fine-tuning sets [25]. Such phenomenon is referred as catastrophic forgetting [46, 18] in fine-tuning [25, 37, 81]. Methods to address catastrophic forgetting include pre-trained weight decay [8, 10, 81], learning rate decreasing [25], and Mixout regularization [37]. These methods focus on parameter space to constrain the distance between two models while our method addresses forgetting following an information-theoretical perspective. In continuous learning, method like rehearsal-based [42, 56, 39, 1] and regularization-based [34, 14, 26] also aims at the forgetting problem, but under a different setting: they focus on balanced performance on tasks with previous data, while in our setting the pre-training corpora are not available and language modeling is not our concern.
Contrastive Learning and Mutual Information: Contrastive learning [49, 23, 61] has demonstrated its power in learning self-supervised representations [9, 21]. The contrastive loss such as InfoNCE[49], InfoMax [23], is proposed as maximizing the mutual information of features in different views, and further discussed in [53, 67, 69, 66, 74]. This paper shares the same perspective of information theory, but looks into a different problem, addressing forgetting for fine-tuning. Addressing forgetting motivates our objective of maximizing the mutual information between the outputs of two models, which is different from conventional contrastive learning, and cooperation with fine-tuning motivates the use of conditional mutual information, which is quite different also. Fischer and Alemi [16] propose to restrict information for robustness in vision domain. Restricting information to some extent helps a model ignore spurious features, but may be at the cost of ignoring robust features as well. In contrast, the proposed method aims to retain the generic and robust features that are already captured by a pre-trained language model, and thus better fits a fine-tuning scenario.

heading: 5 Discussion and Conclusion

text: In this paper, we proposed RIFT to fine-tune a pre-trained language model towards robust down-stream performance. RIFT addresses the forgetting problem from an information-theoretical perspective and consistently outperforms state-of-the-arts in our experiments. We hope that this work can contribute to the NLP robustness in general and thus to more reliable NLP systems in real-life applications.

heading: Acknowledgements

text: This work is supported by the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 (S21/20) and Tier 2.

title: How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?