summaries: The paper solves a problem similar to Neural Body (reconstruction / free-view rendering of clothed humans from videos based on SMPL fits) but does not require training a neural network per scene; instead, it generalises from training sequences and, at test time, runs only a feed-forward model conditioned on keyframes. This is important for many applications since training a neural net per scene is not practical, and also the proposed extension is not trivial: the paper brings novel ideas (using attention for time-wise and camera-wise aggregation). Experimental evaluation is comprehensive and the quality is even superior to per-scene optimised Neural Bodies. Given this, I suggest to accept the paper, although there is some room for improvement.

summaries: This paper presents a method to synthesize a novel view image of a person given the sparse multiview images. The paper proposes time-augmented skeletal features---the pixel-aligned features that encode NeRF-like rendering fields by aggregating image features across  time and views, parametrized by a human mesh model. They use transformer networks to learn fusing multiview image features over time. The resulting features are used to generate an implicit function that can predict RGB and opacity. They demonstrate that the proposed method is generalizable to unseen identities and poses, validated on ZJU-MoCap and AIST datasets.

summaries: This work proposes a method that can synthesize free-viewpoint videos for dynamic humans in a sparse multi-camera system.  As is claimed and demonstrated in the paper, by introducing transformer modules to aggregate spatio-temporal information in the sparse multi-view data, the learned neural radiance fields are generalizable to unseen human identities and unseen poses.  The proposed method is tested on public multi-camera datasets, showing promising results.

summaries: This paper presents a generalizable NeRF network that can produce novel view synthesis for novel dynamic human scenes from sparse camera views, in a feed-forward manner. It leverages visual features from tracked parametric body model and similar to PixelNeRF, it conditions the NeRF representation on pixel-aligned image features for appearance generalization. A novel combination of a temporal Transformer and a multiview Transformer is proposed to integrate multi-frame and multi-view visual observations. It demonstrated stronger generalization capability than prior work.

abstractText: In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, is highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture. Specifically, we first introduce a temporal transformer that aggregates tracked visual features based on the skeletal body motion over time. Moreover, a multi-view transformer is proposed to perform cross-attention between the temporally-fused features and the pixel-aligned features at each time step to integrate observations on the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses. The video results and code are available at https://youngjoongunc.github.io/nhp.

authors:
Kara-Ali Aliev
Dmitry Ulyanov
Victor Lempitsky

title: Neural point-based graphics

venue: arXiv preprint arXiv:1906.08240,

authors:
Nicolas Carion
Francisco Massa
Gabriel Synnaeve
Nicolas Usunier
Alexander Kirillov
Sergey Zagoruyko

title: End-to-end object detection with transformers

venue: In European Conference on Computer Vision,

authors:
Joel Carranza
Christian Theobalt
Marcus A Magnor
Hans-Peter Seidel

title: Free-viewpoint video of human actors

venue: ACM transactions on graphics (TOG),

authors:
Alvaro Collet
Ming Chuang
Pat Sweeney
Don Gillett
Dennis Evseev
David Calabrese
Hugues Hoppe
Adam Kirk
Steve Sullivan

title: High-quality streamable free-viewpoint video

venue: ACM Transactions on Graphics (ToG),

authors:
Edilson De Aguiar
Carsten Stoll
Christian Theobalt
Naveed Ahmed
Hans-Peter Seidel
Sebastian Thrun

title: Performance capture from sparse multi-view video

venue: In ACM SIGGRAPH

authors:
Paul Debevec
Tim Hawkins
Chris Tchou
Haarm-Pieter Duiker
Westley Sarokin
Mark Sagar

title: Acquiring the reflectance field of a human face

venue: In Proceedings of the 27th annual conference on Computer graphics and interactive techniques,

authors:
Mingsong Dou
Sameh Khamis
Yury Degtyarev
Philip Davidson
Sean Ryan Fanello
Adarsh Kowdle
Sergio Orts Escolano
Christoph Rhemann
David Kim
Jonathan Taylor

title: Fusion4d: Real-time performance capture of challenging scenes

venue: ACM Transactions on Graphics (TOG),

authors:
John Flynn
Michael Broxton
Paul Debevec
Matthew DuVall
Graham Fyffe
Ryan Overbeck
Noah Snavely
Richard Tucker

title: Deepview: View synthesis with learned gradient descent

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Guy Gafni
Justus Thies
Michael Zollhofer
Matthias Nießner

title: Dynamic neural radiance fields for monocular 4d facial avatar reconstruction

authors:
Juergen Gall
Carsten Stoll
Edilson De Aguiar
Christian Theobalt
Bodo Rosenhahn
Hans-Peter Seidel

title: Motion capture using joint skeleton tracking and surface estimation

venue: IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Chen Gao
Yichang Shih
Wei-Sheng Lai
Chia-Kai Liang
Jia-Bin Huang

title: Portrait neural radiance fields from a single image

venue: arXiv preprint arXiv:2012.05903,

authors:
Ke Gong
Xiaodan Liang
Yicheng Li
Yimin Chen
Ming Yang
Liang Lin

title: Instance-level human parsing via part grouping network

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Kaiwen Guo
Peter Lincoln
Philip Davidson
Jay Busch
Xueming Yu
Matt Whalen
Geoff Harvey
Sergio Orts-Escolano
Rohit Pandey
Jason Dourgarian

title: The relightables: Volumetric performance capture of humans with realistic relighting

authors:
Youngjoong Kwon
Stefano Petrangeli
Dahun Kim
Haoliang Wang
Henry Fuchs
Viswanathan Swaminathan

title: Rotationally-consistent novel view synthesis for humans

venue: In Proceedings of the 28th ACM International Conference on Multimedia,

authors:
Youngjoong Kwon
Stefano Petrangeli
Dahun Kim
Haoliang Wang
Eunbyung Park
Viswanathan Swaminathan
Henry Fuchs

title: Rotationally-temporally consistent novel view synthesis of human performance video

authors:
Ruilong Li
Shan Yang
David A Ross
Angjoo Kanazawa

title: Ai choreographer: Music conditioned 3d dance generation with aist++

venue: In International Conference on Computer Vision,

authors:
Tianye Li
Mira Slavcheva
Michael Zollhoefer
Simon Green
Christoph Lassner
Changil Kim
Tanner Schmidt
Steven Lovegrove
Michael Goesele
Zhaoyang Lv

title: Neural 3d video synthesis

venue: arXiv preprint arXiv:2103.02597,

authors:
Yiyi Liao
Katja Schwarz
Lars Mescheder
Andreas Geiger

title: Towards unsupervised learning of generative models for 3d controllable image synthesis

authors:
Baoyuan Liu
Min Wang
Hassan Foroosh
Marshall Tappen
Marianna Pensky

title: Sparse convolutional neural networks

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Lingjie Liu
Jiatao Gu
Kyaw Zaw Lin
Tat-Seng Chua
Christian Theobalt

title: Neural sparse voxel fields

venue: arXiv preprint arXiv:2007.11571,

authors:
Lingjie Liu
Weipeng Xu
Michael Zollhoefer
Hyeongwoo Kim
Florian Bernard
Marc Habermann
Wenping Wang
Christian Theobalt

title: Neural rendering and reenactment of human actor videos

authors:
Shaohui Liu
Yinda Zhang
Songyou Peng
Boxin Shi
Marc Pollefeys
Zhaopeng Cui

title: Dist: Rendering deep implicit signed distance function with differentiable sphere tracing

authors:
Stephen Lombardi
Tomas Simon
Jason Saragih
Gabriel Schwartz
Andreas Lehrmann
Yaser Sheikh

title: Neural volumes: Learning dynamic renderable volumes from images

authors:
Matthew Loper
Naureen Mahmood
Javier Romero
Gerard Pons-Moll
Michael J Black

title: Smpl: A skinned multi-person linear model

authors:
Lars Mescheder
Michael Oechsle
Michael Niemeyer
Sebastian Nowozin
Andreas Geiger

title: Occupancy networks: Learning 3d reconstruction in function space

authors:
Ben Mildenhall
Pratul P Srinivasan
Matthew Tancik
Jonathan T Barron
Ravi Ramamoorthi
Ren Ng

title: Nerf: Representing scenes as neural radiance fields for view synthesis

authors:
Ryota Natsume
Shunsuke Saito
Zeng Huang
Weikai Chen
Chongyang Ma
Hao Li
Shigeo Morishima

title: Siclope: Silhouette-based clothed people

authors:
Michael Niemeyer
Lars Mescheder
Michael Oechsle
Andreas Geiger

title: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision

authors:
Kyle Olszewski
Sergey Tulyakov
Oliver Woodford
Hao Li
Linjie Luo

title: Transformable bottleneck networks

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

authors:
Keunhong Park
Utkarsh Sinha
Jonathan T Barron
Sofien Bouaziz
Dan B Goldman
Steven M Seitz
Ricardo-Martin Brualla

title: Deformable neural radiance fields

venue: arXiv preprint arXiv:2011.12948,

authors:
Keunhong Park
Utkarsh Sinha
Peter Hedman
Jonathan T Barron
Sofien Bouaziz
Dan B Goldman
Ricardo Martin-Brualla
Steven M Seitz

title: Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields

venue: arXiv preprint arXiv:2106.13228,

authors:
Sida Peng
Junting Dong
Qianqian Wang
Shangzhan Zhang
Qing Shuai
Xiaowei Zhou
Hujun Bao

title: Animatable neural radiance fields for modeling dynamic human bodies

authors:
Sida Peng
Yuanqing Zhang
Yinghao Xu
Qianqian Wang
Qing Shuai
Hujun Bao
Xiaowei Zhou

title: Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans

venue: In CVPR,

authors:
Songyou Peng
Michael Niemeyer
Lars Mescheder
Marc Pollefeys
Andreas Geiger

title: Convolutional occupancy networks

venue: arXiv preprint arXiv:2003.04618,

authors:
Albert Pumarola
Enric Corona
Gerard Pons-Moll
Francesc Moreno-Noguer

title: D-nerf: Neural radiance fields for dynamic scenes

venue: arXiv preprint arXiv:2011.13961,

authors:
Amit Raj
Michael Zollhoefer
Tomas Simon
Jason Saragih
Shunsuke Saito
James Hays
Stephen Lombardi

title: Pva: Pixel-aligned volumetric avatars

venue: arXiv preprint arXiv:2101.02697,

authors:
Shunsuke Saito
Zeng Huang
Ryota Natsume
Shigeo Morishima
Angjoo Kanazawa
Hao Li

title: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization

authors:
Shunsuke Saito
Tomas Simon
Jason Saragih
Hanbyul Joo

title: Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization

authors:
Shaoshuai Shi
Chaoxu Guo
Li Jiang
Zhe Wang
Jianping Shi
Xiaogang Wang
Hongsheng Li

title: Pv-rcnn: Point-voxel feature set abstraction for 3d object detection

authors:
Vincent Sitzmann
Justus Thies
Felix Heide
Matthias Nießner
Gordon Wetzstein
Michael Zollhofer

title: Deepvoxels: Learning persistent 3d feature embeddings

authors:
Vincent Sitzmann
Michael Zollhöfer
Gordon Wetzstein

title: Scene representation networks: Continuous 3d-structure-aware neural scene representations

venue: arXiv preprint arXiv:1906.01618,

authors:
Carsten Stoll
Juergen Gall
Edilson De Aguiar
Sebastian Thrun
Christian Theobalt

title: Video-based reconstruction of animatable human characters

authors:
Zhuo Su
Lan Xu
Zerong Zheng
Tao Yu
Yebin Liu

title: Robustfusion: Human volumetric capture with data-driven visual cues using a rgbd

authors:
Justus Thies
Michael Zollhöfer
Matthias Nießner

title: Deferred neural rendering: Image synthesis using neural textures

authors:
Shuhei Tsuchida
Satoru Fukayama
Masahiro Hamasaki
Masataka Goto

title: Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing

venue: In ISMIR,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you need

venue: arXiv preprint arXiv:1706.03762,

authors:
Qianqian Wang
Zhicheng Wang
Kyle Genova
Pratul Srinivasan
Howard Zhou
Jonathan T. Barron
Ricardo Martin-Brualla
Noah Snavely
Thomas Funkhouser

title: Ibrnet: Learning multi-view image-based rendering

authors:
Xiaolong Wang
Ross Girshick
Abhinav Gupta
Kaiming He

title: Non-local neural networks

authors:
Minye Wu
Yuehao Wang
Qiang Hu
Jingyi Yu

title: Multi-view neural human rendering

authors:
Wenqi Xian
Jia-Bin Huang
Johannes Kopf
Changil Kim

title: Space-time neural irradiance fields for free-viewpoint video

venue: arXiv preprint arXiv:2011.12950,

authors:
Alex Yu
Vickie Ye
Matthew Tancik
Angjoo Kanazawa

title: pixelnerf: Neural radiance fields from one or few images

authors:
Wentao Yuan
Zhaoyang Lv
Tanner Schmidt
Steven Lovegrove

title: Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering

venue: arXiv preprint arXiv:2101.01602,

authors:
Zerong Zheng
Tao Yu
Yixuan Wei
Qionghai Dai
Yebin Liu

title: Deephuman: 3d human reconstruction from a single image

authors:
Tinghui Zhou
Richard Tucker
John Flynn
Graham Fyffe
Noah Snavely

title: Stereo magnification: Learning view synthesis using multiplane images

venue: arXiv preprint arXiv:1805.09817,

heading: 1 Introduction

text:
Free-viewpoint video of a human performer has a variety of applications in the area of telepresence, mixed reality, gaming and etc. Conventional free-viewpoint video systems require extremely expensive setups such as dense camera rigs [4, 7, 43] or accurate depth sensors [6, 13], to capture person-specific appearance information. In this paper, we aim at a scalable solution for free-viewpoint human performance rendering that generalizes across different human performers and requires only sparse camera views. However, representing and rendering arbitrary human performances is extremely challenging when the observations are highly sparse (up to three to four views) due to heavy self-occlusions and dynamic articulations of the body parts. In particular, an effective solution needs to coherently aggregate appearance information from sparse multi-view observations across time as the body undergoes a 3D motion. Furthermore, the solution needs to generalize to unseen motions and characters at test time.
Recently, neural radiance fields (NeRF) [26, 11, 17, 30, 33, 35, 36, 47, 50, 52, 53] have shown photo-realistic novel view synthesis results in per-scene optimization settings. To avoid the expensive per-scene training and improve the practicality, generalizable NeRFs [36, 52, 47] have been proposed which use image-conditioned, pixel-aligned features and achieve feed-forward view synthesis from sparse input views [36, 52]. Direct application of these methods to complex and non-rigid human motion is not straightforward, however, and naive solutions suffer from significant artifacts as shown in Fig. 3. Some existing methods [37, 52] aggregate image features across multiple views by simple average pooling, which often leads to over-smoothed outputs when details observed from multiple
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
views (e.g., front and side views) differ due to self occlusions of humans. Alternatively, several methods [23, 33] have proposed to learn person-specific global appearance features from multi-view observations. However, such methods are not able to generalize to new human performers.
To address these challenges, we propose Neural Human Performer, a novel approach that learns generalizable radiance fields based on a parametric 3D body model for robust performance capture. In addition to exploiting a parametric body model as a geometric prior, the core of our method is a combination of temporal and multi-view transformers which help to effectively aggregate spatio-temporal observations to robustly compute the density and color of a query point. First, the temporal transformer aggregates trackable visual features based on the input skeletal body motion across time. The following multi-view transformer performs cross-attention between the temporallyaugmented skeletal features and the pixel-aligned features for each time step. The proposed modules collectively contribute to the adaptive aggregation of multi-time and multi-view information, resulting in significant improvements in synthesis results in different generalization settings of unseen motions and identities.
We study the efficacy of Neural Human Performer on two multi-view human performance capture datasets, ZJU-MoCap [33] and AIST [16]. Experiments show that our method significantly outperforms recent generalizable radience field (NeRF) methods. Furthermore, we compare ours with identity-specific methods [33, 44, 49] that also utilize a 3D human body model prior. Surprisingly, our generalized method achieves better rendering quality than the person-specific dedicated methods when tested on novel poses demonstrating the effectiveness of our transformer-based generalizable representation.
To summarize, our contributions are:
• We present a new feed-forward method of synthesizing novel-view videos of arbitrary human performers from sparse camera views. We propose Neural Human Performer that learns generalizable neural radiance representations by leveraging a 3D body motion prior.
• We design a combination of temporal and multi-view transformers that can aggregate information on the fly over video frames across multiple views to render each frame of the novel-view video.
• We show significant improvements over recent generalizable NeRF methods on unseen identities and poses. Moreover, our generalization results can outperform even personspecific methods when tested on unseen poses.

heading: 2 Related works

text:
Human performance capture. Novel view synthesis of human performance has a long history in computer vision and graphics. Traditional methods rely on complicated hardware such as dense camera rigs [4, 7, 43] or accurate depth sensors [6, 13]. To enable free-view video from sparse camera views, template-based methods [3, 5, 10, 42] exploit pre-scanned human models to track the motion of a person. However, their synthesis results are not photo-realistic and pre-scanned human models are not available in most cases. Recent methods [27, 37, 38, 54] learn 3D human geometry priors along with pixel aligned features to enable detailed 3D human reconstructions even from single images. However, these methods often suffer under complex human poses that are never seen during training and hence cannot be directly used for our purpose of human performance synthesis.
Neural 3D representations. Recently, there has been great progress in learning neural networks to represent the shape and appearance of scenes. The 3D representations are learned from 2D images via differentiable rendering networks. Convolutional neural networks are used to predict volumetric representations via 3D voxel-grid features [40, 23, 29, 25, 14, 15], point clouds [1, 49], textured meshes [18, 21, 44] and multi-plane images [8, 55]. The learnt representations are projected by a 3D-to-2D operation to synthesize images. Some methods [14, 15] introduce consistency loss to improve the multi-view and temporal consistency. However, these methods often have difficulty in scaling to higher resolution due to memory restrictions.
To eschew these problems, implicit function-based methods [20, 22, 28, 41] learn a multi-layer perceptron that directly translates a 3D positional feature into a pixel generator. The more recent NeRF [26] learns implicit fields of density and color with a volume rendering technique and achieves photo-realistic view synthesis. Among many following NeRF extensions, [30, 31, 35, 53, 17, 50]
focus on dynamic scenes. While showing impressive results, it’s an extremely under-constrained problem to jointly learn NeRF and highly dynamic deformation fields. To regularize the training, Neural Body [33] combines NeRF with a deformable human body model (e.g., SMPL [24]). Despite the promising results, these general deformable NeRF [17, 53] and human-specific NeRF [11, 9, 33, 32] methods must be optimized for each new video separately, and generalize poorly on unseen scenarios. Generalizable NeRFs [36, 47, 52] try to avoid the expensive per-scene optimization by image-conditioning using pixel-aligned features. However, directly extending such methods to model complex and dynamic 3D humans is not straightforward when available observations are highly sparse. Unlike existing works, our method exploits temporal and multi-view information on-the-fly and achieves free-viewpoint human rendering in a feed-forward manner, also generalizing to new, unseen human identities and poses.

heading: 3 Method

heading: 3.1 Neural Human Performer

text:
Problem definition. In our setting, given a sparse set (e.g., 3 or 4) of multi-view cameras c = 1, . . . , C, input videos of an arbitrary human performance Ic,1:T := {Ic,1, Ic,2, ..., Ic,T } are captured for each camera view c defined by {Kc, [R|t]c}. We assume that the 3D human body model fit corresponding to each frame is given. Our goal is to synthesize a novel view video Îq,1:T for a query viewpoint q defined by {Kq, [R|t]q}.
Overview. To compensate for the sparsity of available input views, we propose to exploit temporal information across video frames. In practice, we sample M memory frames from the original input videos to augment each queried timestep t. Our goal is to learn generalizable 3D representations of human performers from multi-time (M ) and multi-view (C) observations.
To this end, the Neural Human Performer is proposed with two main components as illustrated in Fig. 1. First, we construct the time-augmented skeletal features {s}. We exploit a human body model (SMPL [24]) to construct 3D skeletal features by projecting all the SMPL vertices onto each memory frame and picking up the pixel-aligned image features [36, 37, 52] at the projected 2D locations. The skeletal features are sampled from all memory frames to construct the skeletal feature bank. Inspired by Transformers[2, 46, 48], we propose a temporal Transformer that aggregates these memory features into the time-augmented skeletal features {s′}. In the second stage, given a query 3D point x, skeletal features are sampled at x. In addition, pixel-aligned features {p} at each time t are sampled by directly projecting x onto the input images {I}t. The multi-view Transformer is proposed to learn the correlation between the pixel-aligned features {p} and the time-augmented skeletal features {s′}, and to adaptively fuse the multi-view
information. The final fused feature of the query point x is fed into the radiance field module to predict its color and density.

heading: 3.2 Construction of time-augmented skeletal features

text:
Unlike static scenes, video inputs of moving characters inherently contain more visual cues as the occluded regions in a frame may be visible in other (potentially distant) frames. To take advantage of this temporal information, we first build up the skeletal feature bank from memory frames by leveraging a parametric body model (see Fig. 1). Then, we propose a temporal Transformer module that aggregates such collected features.
For each view c and time t, we first build frame-level skeletal features sc,t ∈ RL×d by sampling image features at the projection of the SMPL vertices in each image Ic,t ∈ RH×W×3. L denotes the number of SMPL vertices and d is the dimension of image features.
After collecting all the skeletal features from all memory frames, we aggregate the time information in an attention-aware manner, instead of using simple average pooling. Specifically, for any ith skeletal feature vertex sic,t ∈ Rd, the proposed temporal Transformer casts attention over all other features contained in the vertex’s memory bank sic,t1:tM = {s i c,t1 , s i c,t2 , ..., s i c,tM } ∈ R
M×d. In particular, we compute soft weights for all memory feature vertices in a non-local manner with respect to the current timestep t. Then, the values of the memory features are weighted summed as
t_attic,t = ψ ( 1√
d0 q(sic,t) · k(sic,t1:tM )
T ) , t_attic,t ∈ R1×M
s′c,t i = t_attic,t · v(sic,t1:tM ) + s i c,t, s ′ c,t ∈ RL×d ∀i
(1)
where ψ represents the softmax operator along the second axis, q(·), k(·) and v(·) are learnable query, key and value embedding functions Rd→d0 of the temporal Transformer. In other words, the representation sic,t of each skeletal vertex at time t is computed through a dynamically weighted combination of all its previous and next representations in the memory frames. This allows our network to incorporate helpful information and ignore irrelevant ones from other timesteps. In practice, the temporal Transformer operation in Eq. (1) is performed by a batch matrix multiplication for all skeletal vertices L and all available viewpoints C.

heading: 3.3 Multi-view aggregation of skeletal and query features

text:
Given a query 3D point x ∈ R3, we retrieve the corresponding (time-augmented) skeletal feature s′c,t
x ∈ Rd at the queried location via trilinear interpolation in the SMPL space with SparseConvNet [19], following [39, 34, 51, 33].
In addition, we sample pixel-aligned image feature pxc,t via direct R3→2 projection of the query point x on Ic,t. It is important to note that the pixel-aligned feature pxc,t is time-specific and represents the exact query location of x, while the skeletal feature s′c,t
x is time-augmented (w.r.t t) and contains inherent geometric deviations in the SMPL vertices and the following trilinear interpolations. We propose to combine these two complementary features, which will be shown to be effective in Sec. 4.4.
Given the two sets of multi-view features, skeletal s′1:C,t x = {s′c,t x|c = 1, ..., C} ∈ RC×d and pixel-aligned px1:C,t = {pxc,t|c = 1, ..., C} ∈ RC×d, we propose a multi-view Transformer that performs cross-attention from skeletal to pixel-aligned features. Specifically, the values of pixelaligned features from all viewpoints is re-weighted based on how much compatible they are with each skeletal features. The non-local cross-attention mv_att is constructed as:
mv_attxt = ψ ( 1√
d1 k(s′1:C,t x ) · k(px1:C,t)T
) , mv_attxt ∈ RC×C
zx1:C,t = mv_att x t · v(px1:C,t) + v(s′1:C,t x ), zx1:C,t ∈ RC×d, zxc,t ∈ R1×d
(2)
where ψ represents the softmax operator along the second axis. Note that k and v are new layers independent from those in the temporal Transformer. The confident observations in each view will have large weights and be highlighted, and vice versa. Finally, we use the view-wise mean of zxt = 1 C ∑ c z x c,t ∈ Rd as our meta-time and meta-view representation of the query point x.
The final density σt(x) and color values rgbt(x) at time t are computed as:
σt(x) =MLPσ(z x t ), rgbt(X) =MLPrgb( ∑ c (zxc,t; γd(d))/C), (3)
where MLPσ and MLPrgb consist of four and two linear layers respectively, and γd : R3→6×l is a positional encoding of viewing direction d ∈ R3 as in [26] with 2× l different basis functions. More details on the network architecture can be found in the supplementary material.

heading: 3.4 Volume Rendering

text:
The predicted color of a pixel p ∈ R2 for a target viewpoint q in the focal plane of the camera and center r0 ∈ R3 is obtained by marching rays into the scene using the camera-to-world projection matrix, P−1 = [Rq|tq]−1K−1q with the direction of the rays given by d = P−1p−r0 ∥P−1p−r0∥ .
We then accumulate the radiance and opacity along the ray r(z) = r0 + zd for z ∈ [znear, zfar] as defined in NeRF [26] as follows:
Iq(p) = ∫ zfar znear T(z)σ(r(z))c(r(z),d)dz, where T(z) = exp ( − ∫ z znear σ(r(s))ds ) (4)
In practice, we uniformly sample a set of 64 points z ∼ [znear, zfar]. We set X = r(z) and use the quadrature rule to approximate the integral. We compute the 3D bounding box of the SMPL parameters at time t and derive the bounds for ray sampling znear, zfar.

heading: 3.5 Loss Function

text: Given the ground truth target images Iq,t, we train both the radiance field and feature extraction networks using a simple photometric reconstruction loss L = ∥Îq,t − Iq,t∥2 .

heading: 4 Experiments

text: We present novel view synthesis and 3d reconstruction results of human performances in different generalization scenarios. We compare our method against the current best view-synthesis methods from two classes: per-subject optimization methods that also use a human body model prior (Sec. 4.1) and generalizable NeRF methods (Sec. 4.2). We experiment on the ZJU-MoCap [33] and AIST datasets [45, 16]. For training and testing our model as well as the baselines, we remove the background using the foreground masks that are either provided by the dataset or pre-computed using an off-the-shelf method. Unless otherwise specified, we sample two memory frames {t− 20, t+ 20} at time t (i.e., three timesteps in total) and take three canonical input views in all experiments. The details of the implementation, datasets, training process are provided in the supplementary material.

heading: 4.1 Comparison with body model-based, per-subject optimization methods.

text:
Baselines. We perform comparisons with the state-of-the-art Neural Body (NB) [33] that combines the body model prior SMPL and NeRF in a per-subject optimization setting. Neural Textures (NT) [44] renders a coarse mesh with latent texture maps and uses a 2D CNN to render target images. We use the SMPL mesh as the coarse mesh input to NT in our experiments. NHR [49] extracts 3D features from input point clouds and renders them into 2D images. Since dense point clouds are difficult to obtain from sparse camera views, we use the SMPL vertices as input point clouds. These methods have reported that they can adapt to new poses of the same performer, hence we compare to them for the task of novel pose synthesis.
Setup. We experiment with ZJU-MoCap dataset [33] which provides performance captures of 10 human subjects captured from 23 synchronized cameras, human body model parameters as well as the foreground mask corresponding to each frame. Each video contains complex motions such as kicking and Taichi and is between 1000 to 2000 frames long. We first split the dataset into two parts: source and target videos. In all comparisons, the first 300 frames of either source or target videos are used during training, and the remaining next frames (unseen poses) are held out for testing. Note that
Method PSNR SSIM Trained on source subjects NB 23.79 0.887 NHR 22.31 0.871 NT 22.28 0.872 Trained on source subjects Ours 26.94 0.929
a . Test results on unseen poses of source subjects
Method PSNR SSIM Trained on target subjects NB 22.88 0.880 NHR 22.03 0.875 NT 21.92 0.873 Trained on source subjects Ours 24.75 0.906
b . Test results on unseen poses of target subjects
Method PSNR SSIM Trained on source subjects NB 28.51 0.947 NHR 23.95 0.897 NT 23.86 0.896 Trained on source subjects Ours 28.73 0.936 Ours per-subject 31.57 0.966
c . Test results on seen poses of source subjects
Table 1: Comparison with other body model-based, per-subject optimization methods.
Figure 2: Pose generalization – comparison with per-scene optimization methods that use a human body prior. Results of NT(Neural textures) [44], NHR (Neural human rendering) [49], NB (Neural body) [33] and ours. Novel view synthesis on the ZJU-MoCap for unseen poses of source subjects. While ours is a single model trained on all source subjects; baselines are trained in a per-subject manner.
the baseline methods are always trained in a per-subject manner. To validate whether the training is reproducible, we experiment with 5 independent runs with random train/test splits and observe a variance of 0.15 PSNR, showing that the results are consistent. In each independent run, we used seven models for training and the other three for testing.
Results. We present three different comparison settings to validate our method. We would like to point out that all the comparison settings place our method (‘ours’) in disadvantage. This is because our model is trained on all the source subjects at once (one network for all subjects), while the competing methods are per-subject trained on the subject to be tested (one network for one subject) - easier setting. First, we evaluate the pose generalization capability (Table. 1a and Fig. 2) where we train on the training splits of the source subjects and test on the testing splits, i.e., unseen poses, of the same source subjects. Our method significantly outperforms all the baselines and the state-of-the-art Neural Body [33] by +3.15 PSNR and +0.042 SSIM scores. We next evaluate the identity generalization capability (Table. 1b) by testing on the unseen poses of target subjects. While our method is trained on source subjects only, baselines are trained on target subjects. Note that this comparison is disadvantageous to us since unlike our method the baselines have seen the testing subjects as they must be trained separately for each subject. Surprisingly, however, our method still outperforms all the baselines by a healthy margin of +1.87 PSNR and +0.026 SSIM scores. These results indicate that our proposed architecture with the temporal and the multi-view transformers generalizes well across unseen identities and poses, and produces photo-realistic results. Finally, in Table. 1c, we evaluate our method and baselines on seen poses of seen subjects. Even when we train a single model of our method for all the source sobjects (‘ours’), ours achieves comparable results with the state-of-the-art per-subject based methods [33]. To make a more fair comparison, we also train our method in the same per-subject setting similar to the baselines. Table. 1c shows that in this case, our model (‘ours per-subject’) achieves a significant improvement over the best-performing baseline [33] by +3 PSNR and +1.4% SSIM.

heading: 4.2 Comparison with generalizable NeRF methods.

text:
Baselines. Among the recent generalizable NeRF methods [36, 52, 47], we compare with PixelNeRF [52] and PVA [36] which focus on very sparse (up to 3 or 4) input views. We reimplement [36] since the code is not available.
Setup. In addition to the ZJU-MoCap dataset (see Sec. 4.1), we experiment on the larger AIST dataset [45, 16] to further evaluate the generalization abilities of different methods. AIST dataset provides dance videos of 30 human subjects captured from 9 cameras, together with the corresponding SMPL fits. We extract the foregroud mask of each image using an off-the-shelf human parser [12]. AIST dataset contains highly diverse motions, slow to fast, simple to complex. We split the dataset into 20 and 10 subjects for training and testing respectively, where the testing dataset contains unseen poses of unseen subjects.
Novel view synthesis results. As shown in Table. 2, across all the datasets and all the metrics, our method consistently outperforms the baselines by healthy margins of +1.6 PSNR and +0.037 SSIM scores. Fig. 3 and Fig. 5 present the same tendency in visualizations. Pixel-NeRF and PVA aggregate multi-view observations via average pooling without explicitly considering the correlation between the views. In contrast, our temporal and multi-view transformers learn to model the correlation between input views and integrate different observations to help the NeRF module to produce higher quality results. Another advantage of our method is that the used body model prior provides a robust geometric cue to handle the self-occlusion of the human body.
3D reconstruction results. We also evaluate the 3D reconstruction quality of generalizable NeRF methods and our method on the ZJU-MoCap (Fig. 4) and AIST datasets (Fig. 5) given three input views. The visualizations show that our 3D reconstructions align well with the input images, and is more accurate than even the per-subject method [33] (e.g., the shape of the upper cloth in Fig. 4).
Overall, these results verify that since humans are complex and prone to occlusions, more sophisticated designs compared to simple image-conditioning are required to learn robust and accurate appearance representations.

heading: 4.3 Cross-dataset generalization.

text:
We further study the generalization ability of our method across different datasets by training on one dataset and testing on another one as shown in Table. 4a. We would like to point out that the two
datasets [33, 16] have significantly different statistics both in terms of color distribution (background, lighting) and distance of the camera to the subject, making the cross-dataset generalization task extremely challenging. Nevertheless, we found that only 8-minute fine-tuning on the target dataset can already outperform the baselines fully-trained on the target dataset. 16-minute fine-tuning performs on par with our model fully-trained on the target dataset.

heading: 4.4 Ablation studies

text:
Table. 3 shows the ablation study on the ZJU-MoCap dataset for unseen subjects and unseen poses, using three time-steps and three camera views as input. Note that all the variations without either temporal or multi-view transformer modules use simple average pooling instead, to fuse temporal or multi-view observations respectively.
Complementariness of skeletal and pixel-aligned query features. ‘Sk’ uses only timeaugmented skeletal features (Sec. 3.2) without time-specific pixel-aligned features, while ‘Px’ uses only the time-specific pixel-aligned features, on the contrary. Both ‘only’ models show the largest drops compared to our full model, and ‘Sk + Px’ model improves them by +1.2 PSNR and +0.9 PSNR respectively. This validates the complementariness between the skeletal and pixel-aligned features in that one is time-augmented but involves slight geometric deviations, while another is time-specific and represents exact query location, as discussed in Sec. 3.3.
Impact of temporal and multi-view transformers. ‘Sk + Px’ uses no transformers so far, falling behind our full model by -1.3 PSNR score. Then ‘Sk + Px + T’ adds the temporal transformer and improves +0.7 PSNR score, showing its effectiveness in aggregating information over video frames. ‘Sk + Px + MV’ uses the multi-view transformer module and shows the largest gain of +1.0 PSNR, indicating the efficacy of learnt cross-attention between the skeletal features and pixel-aligned features, as well as the importance of learnt inter-view correlations. Our full model ‘Sk + Px + T + MV’ shows the best use of all the proposed components and achieves 24.75 PSNR and 0.9058 SSIM.
Impact of number of camera views. Table. 4b shows the performance of our method when tested with different number of input views, fixing the number of timesteps to one. As expected, the performance degrades slightly with fewer input views (e.g., as few as one).

heading: 5 Limitations

text: While our novel generalizable human performance rendering method outperforms recent per-subject and generalizable NERF methods, there are remaining challenges yet to be explored. 1) While we show that our method can generalize across datasets with finetuning, the generalization capability will be limited as the distribution of the datasets become significantly different. 2) The performance of our method will be affected as the SMPL parameter accuracy degrades. This is the reason for lower performance on the AIST dataset where the accuracy of the SMPL fits are low due to highly complex motion sequences. It is an interesting direction to jointly refine the SMPL parameters using differentiable rendering within our framework. 3) Our algorithm does not have an explicit assumption of static cameras. However, in practice, it might be hard to estimate the inputs to our method (SMPL fits, camera parameters, foreground masks) with moving cameras due to motion blur, changing background, lighting, and synchronization issues etc. We consider this as an orthogonal problem and expect that any advancements in unconstrained multi-view capture setups will help to generalize our method to in-the-wild settings.

heading: 6 Societal impact

text: We discuss the potential societal impact of our work. On a positive note, the human performance synthesis is the key component of realizing telepresence, which has become more important especially in this pandemic era. In the future, people physically apart can feel like they are in the same space and feel connected with a few inexpensive webcams and AR/VR headsets thanks to the development of the telepresence. Like any other technology, our method can also be used for other reasons by corporations, e.g., for identity recognition from a small number of surveillance cameras. We strongly hope that our research will be used in positive directions.

heading: 7 Conclusion

text:
We present Neural Human Performer, a generalizable radiance field network based on a parametric body model that can synthesize free-viewpoint videos for arbitrary human performers from sparse camera views. Leveraging the trackable visual features from the input body motion prior, we propose a combination of a temporal and a multi-view Transformer that integrates multi-time and multi-view observations in a feed-forward manner. Our method can produce photo-realistic view synthesis of unseen poses and identities at test time. In various generalization settings on the ZJU-MoCap and AIST datasets, our method achieves state-of-the-art performance compared to other per-subject methods that utilize a human body prior as well as the generalizable NeRF methods.
Acknowledgments and Disclosure of Funding
We thank Sida Peng of Zhejiang University, Hangzhou, China, for many very helpful discussions on a variety of implementation details of the Neural Body [33]. We thank Ruilong li and Alex Yu of UC Berkeley for many discussions on the AIST++ dataset [16] and Pixel-NeRF [52] details. We thank Prof. Alex Berg of UNC for the generous offer of computational resources and Misha Shvets of UNC for a useful tutorial on it. This work was partially supported by National Science Foundation Award 1840131.

title: Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering


summaries: This paper introduces a neural architecture search method called S3 that first searches the search space before identifying good architectures from the discovered search space.  The authors focus on applying S3 to vision transformers and design a search space modeled after ViT.  They evaluate S3 discovered architectures on various computer vision tasks and show good performance on image classification, object detection, and semantic segmentation.

summaries: This work applies NAS to the vision transformer architecture. Besides searching the architectures, it also automatically adjusts the search space guided by the E-T error that combines the error distribution and top model errors to evaluate the quality of different choices in a search dimension. By fitting a linear function to approximate the E-T error, the new search space is updated through a simple formula. They used the supernet to efficiently compute the E-T errors, and use evolutionary search to search for the best algorithm in a given search space. The discovered architecture is shown to be competitive to the state-of-the-art models, and also generalize to other vision tasks.

summaries: This paper proposes a method for searching the search space of vision. The authors propose decomposing the search space by the search dimensions and evaluating them with the proposed E-T error to explore the optimal search space. Then the new subspace is formulated as Equation (8). The results reported in this paper have achieved state-of-the-art results.

summaries: This work propose to first search for the search space for transformers and then applies once-for-all NAS methods to search for architectures within the searched search space. In particular, the work proposes an E-T error metric to evaluate the quality of the search spaces. From the results of the whole search process, this work also summarizes some design guidelines for vision transformers. The experiments are extensive and they show the superiority of the proposed search method.

abstractText: Vision Transformer has shown great visual representation power in substantial vision tasks such as recognition and detection, and thus been attracting fast-growing efforts on manually designing more effective architectures. In this paper, we propose to use neural architecture search to automate this process, by searching not only the architecture but also the search space. The central idea is to gradually evolve different search dimensions guided by their E-T Error computed using a weight-sharing supernet. Moreover, we provide design guidelines of general vision transformers with extensive analysis according to the space searching process, which could promote the understanding of vision transformer. Remarkably, the searched models, named S3 (short for Searching the Search Space), from the searched space achieve superior performance to recently proposed models, such as Swin, DeiT and ViT, when evaluated on ImageNet. The effectiveness of S3 is also illustrated on object detection, semantic segmentation and visual question answering, demonstrating its generality to downstream vision and vision-language tasks. Code and models will be available at here.

authors:
Andrew Brock
Theodore Lim
James M Ritchie
Nick Weston

title: Smash: one-shot model architecture search through hypernetworks

venue: In ICLR,

authors:
Han Cai
Chuang Gan
Tianzhe Wang
Zhekai Zhang
Song Han

title: Once for all: Train one network and specialize it for efficient deployment

authors:
Han Cai
Ligeng Zhu
Song Han

title: Proxylessnas: Direct neural architecture search on target task and hardware

authors:
Nicolas Carion
Francisco Massa
Gabriel Synnaeve
Nicolas Usunier
Alexander Kirillov
Sergey Zagoruyko

title: End-to-end object detection with transformers

venue: In ECCV,

authors:
Mark Chen
Alec Radford
Rewon Child
Jeffrey Wu
Heewoo Jun
David Luan
Ilya Sutskever

title: Generative pretraining from pixels

venue: In International Conference on Machine Learning,

authors:
Minghao Chen
Houwen Peng
Jianlong Fu
Haibin Ling

title: Autoformer: Searching transformers for visual recognition

authors:
Xin Chen
Lingxi Xie
Jun Wu
Qi Tian

title: Progressive differentiable architecture search: Bridging the depth gap between search and evaluation

venue: In ICCV,

authors:
Yuanzheng Ci
Chen Lin
Ming Sun
Boyu Chen
Hongwen Zhang
Wanli Ouyang

title: Evolving search space for neural architecture search

venue: arXiv preprint arXiv:2011.10904,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

authors:
Alexey Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
Mostafa Dehghani
Matthias Minderer
Georg Heigold
Sylvain Gelly

title: An image is worth 16x16 words: Transformers for image recognition at scale

authors:
Thomas Elsken
Jan Hendrik Metzen
Frank Hutter

title: Neural architecture search: A survey

venue: J. Mach. Learn. Res.,

authors:
Zichao Guo
Xiangyu Zhang
Haoyuan Mu
Wen Heng
Zechun Liu
Yichen Wei
Jian Sun

title: Single path one-shot neural architecture search with uniform sampling

authors:
Kai Han
Yunhe Wang
Hanting Chen
Xinghao Chen
Jianyuan Guo
Zhenhua Liu
Yehui Tang
An Xiao
Chunjing Xu
Yixing Xu

title: A survey on visual transformer

venue: arXiv preprint arXiv:2012.12556,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In CVPR, 2016

authors:
Byeongho Heo
Sangdoo Yun
Dongyoon Han
Sanghyuk Chun
Junsuk Choe
Seong Joon Oh

title: Rethinking spatial dimensions of vision transformers

venue: arXiv preprint arXiv:2103.16302,

authors:
Andrew Howard
Mark Sandler
Grace Chu
Liang-Chieh Chen
Bo Chen
Mingxing Tan
Weijun Wang
Yukun Zhu
Ruoming Pang
Vijay Vasudevan

title: Searching for mobilenetv3

authors:
Yiming Hu
Yuding Liang
Zichao Guo
Ruosi Wan
Xiangyu Zhang
Yichen Wei
Qingyi Gu
Jian Sun

title: Angle-based search space shrinking for neural architecture

authors:
Zhicheng Huang
Zhaoyang Zeng
Yupan Huang
Bei Liu
Dongmei Fu
Jianlong Fu

title: Seeing out of the box: End-to-end pre-training for vision-language representation learning

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

venue: In ICML,

authors:
Salman Khan
Muzammal Naseer
Munawar Hayat
Syed Waqas Zamir
Fahad Shahbaz Khan
Mubarak Shah

title: Transformers in vision: A survey

venue: arXiv preprint arXiv:2101.01169,

authors:
Changlin Li
Tao Tang
Guangrun Wang
Jiefeng Peng
Bing Wang
Xiaodan Liang
Xiaojun Chang

title: Bossnas: Exploring hybrid cnn-transformers with block-wisely self-supervised neural architecture search

venue: arXiv preprint arXiv:2103.12424,

authors:
Liam Li
Ameet Talwalkar

title: Random search and reproducibility for neural architecture search

venue: In UAI,

authors:
Xiang Li
Chen Lin
Chuming Li
Ming Sun
Wei Wu
Junjie Yan
Wanli Ouyang

title: Improving one-shot nas by suppressing the posterior fading

authors:
Tsung-Yi Lin
Michael Maire
Serge Belongie
James Hays
Pietro Perona
Deva Ramanan
Piotr Dollár
C Lawrence Zitnick

title: Microsoft coco: Common objects in context

authors:
Chenxi Liu
Barret Zoph
Maxim Neumann
Jonathon Shlens
Wei Hua
Li-Jia Li
Li Fei-Fei
Alan Yuille
Jonathan Huang
Kevin Murphy

title: Progressive neural architecture search

authors:
Hanxiao Liu
Karen Simonyan
Yiming Yang

title: DARTS: Differentiable architecture search

authors:
Ze Liu
Yutong Lin
Yue Cao
Han Hu
Yixuan Wei
Zheng Zhang
Stephen Lin
Baining Guo

title: Swin transformer: Hierarchical vision transformer using shifted windows

venue: arXiv preprint arXiv:2103.14030,

authors:
Ilya Loshchilov
Frank Hutter

title: Decoupled weight decay regularization

venue: In ICLR, 2019

authors:
Niv Nayman
Asaf Noy
Tal Ridnik
Itamar Friedman
Rong Jin
Lihi Zelnik

title: Xnas: Neural architecture search with expert advice

venue: In NeurIPS,

authors:
Asaf Noy
Niv Nayman
Tal Ridnik
Nadav Zamir
Sivan Doveh
Itamar Friedman
Raja Giryes
Lihi Zelnik

title: Asap: Architecture search, anneal and prune

venue: In AISTATS,

authors:
Zizheng Pan
Bohan Zhuang
Jing Liu
Haoyu He
Jianfei Cai

title: Scalable visual transformers with hierarchical pooling

venue: arXiv preprint arXiv:2103.10619,

authors:
Houwen Peng
Hao Du
Hongyuan Yu
Qi Li
Jing Liao
Jianlong Fu

title: Cream of the crop: Distilling prioritized paths for one-shot neural architecture

venue: search. NeurIPS,

authors:
Juan-Manuel Pérez-Rúa
Moez Baccouche
Stephane Pateux

title: Efficient progressive neural architecture search

venue: In BMVC,

authors:
Hieu Pham
Melody Guan
Barret Zoph
Quoc Le
Jeff Dean

title: Efficient neural architecture search via parameters sharing

authors:
Ilija Radosavovic
Raj Prateek Kosaraju
Ross Girshick
Kaiming He
Piotr Dollár

title: Designing network design spaces

authors:
Esteban Real
Alok Aggarwal
Yanping Huang
Quoc V Le

title: Regularized evolution for image classifier architecture search

venue: In AAAI,

authors:
David So
Quoc Le
Chen Liang

title: The evolved transformer

venue: In International Conference on Machine Learning. PMLR,

authors:
Aravind Srinivas
Tsung-Yi Lin
Niki Parmar
Jonathon Shlens
Pieter Abbeel
Ashish Vaswani

title: Bottleneck transformers for visual recognition

venue: arXiv preprint arXiv:2101.11605,

authors:
Mingxing Tan
Quoc V. Le

title: Efficientnet: Rethinking model scaling for convolutional neural networks

venue: In ICML, 2019

authors:
Hugo Touvron
Matthieu Cord
Matthijs Douze
Francisco Massa
Alexandre Sablayrolles
Herv’e J’egou

title: Training data-efficient image transformers & distillation through attention

venue: arXiv preprint arXiv:2012.12877,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Dilin Wang
Meng Li
Chengyue Gong
Vikas Chandra

title: Attentivenas: Improving neural architecture search via attentive sampling

authors:
Hanrui Wang
Zhanghao Wu
Zhijian Liu
Han Cai
Ligeng Zhu
Chuang Gan
Song Han

title: Hat: Hardware-aware transformers for efficient natural language processing

venue: In ACL,

authors:
Wenhai Wang
Enze Xie
Xiang Li
Deng-Ping Fan
Kaitao Song
Ding Liang
Tong Lu
Ping Luo
Ling Shao

title: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

venue: arXiv preprint arXiv:2102.12122,

authors:
Bichen Wu
Xiaoliang Dai
Peizhao Zhang
Yanghan Wang
Fei Sun
Yiming Wu
Yuandong Tian
Peter Vajda
Yangqing Jia
Kurt Keutzer

title: Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search

authors:
Kan Wu
Houwen Peng
Minghao Chen
Jianlong Fu
Hongyang Chao

title: Rethinking and improving relative position encoding for vision transformer

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),

authors:
Lingxi Xie
Alan Yuille

title: Genetic cnn

authors:
Saining Xie
Ross Girshick
Piotr Dollár
Zhuowen Tu
Kaiming He

title: Aggregated residual transformations for deep neural networks

authors:
Yuhui Xu
Lingxi Xie
Xiaopeng Zhang
Xin Chen
Guo-Jun Qi
Qi Tian
Hongkai Xiong

title: PC-DARTS: Partial channel connections for memory-efficient architecture search

authors:
Antoine Yang
Pedro M Esperança
Fabio M Carlucci

title: Nas evaluation is frustratingly hard

authors:
Shan You
Tao Huang
Mingmin Yang
Fei Wang
Chen Qian
Changshui Zhang

title: Greedynas: Towards fast one-shot nas with greedy supernet

authors:
Jiahui Yu
Pengchong Jin
Hanxiao Liu
Gabriel Bender
Pieter-Jan Kindermans
Mingxing Tan
Thomas Huang
Xiaodan Song
Ruoming Pang
Quoc Le

title: Bignas: Scaling up neural architecture search with big single-stage models

venue: NeurIPS,

authors:
Li Yuan
Yunpeng Chen
Tao Wang
Weihao Yu
Yujun Shi
Francis EH Tay
Jiashi Feng
Shuicheng Yan

title: Tokens-to-token vit: Training vision transformers from scratch on imagenet

venue: arXiv preprint arXiv:2101.11986,

authors:
Peng Zhang
Yash Goyal
Douglas Summers-Stay
Dhruv Batra
Devi Parikh. Yin
Yang

title: Balancing and answering binary visual questions

authors:
Sixiao Zheng
Jiachen Lu
Hengshuang Zhao
Xiatian Zhu
Zekun Luo
Yabiao Wang
Yanwei Fu
Jianfeng Feng
Tao Xiang
Philip HS Torr

title: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers

venue: arXiv preprint arXiv:2012.15840,

authors:
Bolei Zhou
Hang Zhao
Xavier Puig
Sanja Fidler
Adela Barriuso
Antonio Torralba

title: Scene parsing through ade20k dataset

authors:
Barret Zoph
Quoc V Le

title: Neural architecture search with reinforcement learning

heading: 1 Introduction

text:
Vision transformer recently has drawn great attention in computer vision because of its high model capability and superior potentials in capturing long-range dependencies. Building on top of transformers [41], modern state-of-the-art models, such as ViT [10] and DeiT [40], are able to achieve competitive performance on image classification and downstream vision tasks [13, 20] compared to CNN models. There are fast-growing efforts on manually designing more effective architectures to further unveil the power of vision transformer [27, 53, 15].
Neural architecture search (NAS) as a powerful technique for automating the design of networks has shown its superiority to manual design [58, 39]. For NAS methods, search space is crucial because it determines the performance bounds of architectures during the search. It has been observed that the improvement of search space induced a favorable impact on the performance of many state-of-the-art models [2, 16, 50]. Researchers have devoted ample efforts to the design of CNN space [45, 12, 42]. However, limited attention is paid to Transformer counterparts, leading to the dilemma of search space design when finding more effective and efficient transformer architectures.
In this paper, we propose to Search the Search Space (S3) by automatically refining the main changeable dimensions of transformer architectures. In particular, we try to answer the following two key questions: 1) How to effectively and efficiently measure a specific search space? 2) How to automatically change a defective search space to a good one without human prior knowledge?
For the first question, we propose a new measurement called E-T Error for search space quality estimation and utilize a once-for-all supernet for effective and efficient computing. Specifically, the “E” part in the E-T error is similar to the error empirical distribution function used in [35] focusing on
∗Equal contributions. Work done when Minghao, Kan, Bolin are interns of MSRA. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
the overall quality, while the “T” part considers the quality of top-tier architectures in a search space. In contrast to [35] that trains each subnet for only a few epochs, we train a once-for-all supernet with AutoFormer [6] to obtain a large amount of well-trained subnets, providing a more reliable proxy for performance estimation.
For the second question, as shown in Fig. 1 we decompose the search space by multiple dimensions, including depth, embedding dimension, MLP ratio, window size, number of heads and Q-K-V dimension, and progressively evolve each dimension to compose a better space. Particularly, we use linear parameterization to model the tendencies of different dimensions so as to guide the search.
In addition, we analyze the search process of search space and provide some interesting observations and design guidelines: 1) The third stage is the most important one and increasing its number of blocks leads to performance improvement. 2) Shallow layers should use a small window size while deep layers should use a large window size. 3) MLP ratio is better to be progressively increased along with network depth. 4) Q-K-V dimension could be smaller than the embedding dimension without performance drop. We hope these observations will help both the manual design of vision transformer and the space design for automatic search.
In summary, we make the following contributions:
• We propose a new scheme, i.e. S3, to automate the design of search space for vision transformer. We also present a new pipeline for vision transformer search with minimal human interactions. Moreover, we provide analysis and guidelines on vision transformer architecture design, which might be helpful for future architecture search and design.
• The experiments on ImageNet verify the proposed automatic search space design method can improve the effectiveness of design space, thus boosting the performance of searched architectures. The discovered models achieve superior performance compared to the recent ViT [10] and Swin [27] transformer families under aligned settings. Moreover, the experiments on downstream vision and vision-language tasks, such as object detection, semantic segmentation and visual question answering, demonstrate the generality of the method.

heading: 2 Approach

heading: 2.1 Problem Formulation

text:
Most of existing NAS methods can be formulated as a constrained optimization problem as:
α∗A = arg min α∈A L(W ∗α;Dval) s.t. W ∗α = arg min Wα L(Wα;Dtrain), g(α) < c, (1)
whereWα is the weights of the network architecture α,A is a predefined search space, L(·) is the loss function, Dtrain and Dval represent train and validation datasets respectively, g(·) denotes a function calculating resource consumption of a model while c denotes a given resource constraint.
Ideally, the search space A is an infinite space Ω that contains all possible architectures. In practice, A is usually a small subspace of Ω due to limited computation resource. In this paper, we break the
convention of searching in a fixed space by considering the architecture search together with the search space design. More concretely, we decouple the constrained optimization problem of Eq. (1) into three separate steps:
Step 1. Searching for an optimal search space under a specific constraint:
A∗ = min A⊂Ω Q(A;Dval) s.t. |A| ≤M, (2)
where Q is a measurement of search space to be elaborated in Sec. 2.3, and M represents the maximum search space size.
Step 2. Following one-shot NAS [12, 34], we encode the search space into a supernet and optimize its weights. This is formulated as solving the following problem:
W ∗A∗ = arg min W Eα∈A∗ [L(Wα;Dtrain)], (3)
where W is the weights of the supernet, Wα is the weights in W specified by the architecture α.
Step 3. After obtaining the well-trained supernet, we search for the optimal architecture via ranking the performance using the learned weights, which is formulated as:
α∗A∗ = arg min α∈A∗ L(W ∗α;Dval), s.t. g(α) < c, (4)
where W ∗α is the weights of architecture α inherited from W ∗ A∗ .

heading: 2.2 Basic Search Space

text:
We set up the general architectures of vision transformer by following the settings in ViT [10] and Swin-T [27] as shown in Fig. 1 (right). To be specific, given an input image, we uniformly split it into non-overlap patches. Those patches are linearly projected to vectors named patch embeddings. The embeddings are then fed into the transformer encoder, which performs most of the computation. At last, a fully connected layer is adopted for classification.
The transformer encoder consists of four sequential stages with progressively reduced input resolutions. Each stage contains blocks with the same embedding dimension. Therefore, stage i has two search dimensions: the number of blocks di and embedding dimension hi. Each block contains a window-based multi-head self-attention (WSA) module and a feed-forward network (FFN) module. We do not force the blocks in one stage to be identical. Instead, block j in the ith stage have several search dimensions including window sizewij , number of heads n i j , MLP ratiomi,Q-K-V embedding dimension qij .

heading: 2.3 Searching the Search Space

text:
Space Quality Evaluation. For a given space A, we propose to use E-T Error, denoted as Q(A), for evaluating its quality. It is the average of two parts: expected error rate Qe(A) and top-tier error Qt(A). For a given space A, the definition of Qe(A) is given as:
Qe(A) = Eα∈A,g(α)<c[eα], (5)
where eα is the top-1 error rate of architecture α evaluated on ImageNet validation set, g(·) is the computation cost function and c is the computation resource constraint. It measures the overall quality of a search space. In practice, we use the average error rate of N randomly sampled architectures to approximate the expectation term. The top-tier error Qt(A) is the average error rate of top 50 candidates under resource constraints, representing the performance upper bound of the search space.
Once-for-all Supernet. In constrast to RegNet [35] that trains hundreds of models with only a few epochs (∼10) and uses their errors as a performance proxy of the fully trained networks, we encode the search space into a supernet and optimize it similar to AutoFormer [6]. Under such training protocol, we are able to train an once-for-all transformer supernet that serves as a reliable and efficient performance indicator.
Searching the Search Space. The search of search space mainly contains two iterative steps: 1) supernet optimization and 2) space evolution.
Algorithm 1 Searching the Search Space Input: Infinite search space Ω, max iteration T , threshold τ for subspace evolved, the subspaces
S (0) 1 , · · · , S (0) D , evolve steps for subspaces γ1, · · · , γD
Output: The most promising search space A∗ 1: Initialize a search space A(0) from Ω with minimal prior human knowledge 2: while t ∈ [0, 1, · · · , T ] do 3: Optimize the weights W (t)A of supernet corresponding to the space A(t) 4: Randomly sample N architectures from the well converged supernet 5: for space S(t)i in S (t) 1 , · · · , S (t) D do
6: Get the subspace {α|α ∈ A(t) ∩ αi = v(t)l } for each choice v (t) l ∈ S (t) i , where αi is the
ith dimension value of the architecture α 7: Calculate the E-T Error of each subspace according to Eq. (5) 8: Evolve the subspace and get S(t+1)Di according to Eq. (8) 9: end for
10: A∗ = A(t+1) = S(t+1)1 × S (t+1) 2 × · · · × S (t+1) D 11: end while
Step 1. For the search space A(t) of tth iteration, we encode it into a supernet. We adopt sandwich training [52] to optimize the weights of supernet, i.e., sampling the largest, the smallest, and two middle models for each iteration and fusing their gradients to update the supernet. Notably, we do not use inplace-distill since it leads to collapse for transformer training.
Step 2. We first decompose the search space A(t) by the search dimensions, defined in Sec .2.2, and four stages. Denote their corresponding subspaces as S(t)1 , S (t) 2 , · · · , S (t) D , where D is the product of the number of stage and the number of search dimensions. The search space A(t) now can be expressed as the Cartesian product of all search spaces:
A(t) = S(t)1 × S (t) 2 × · · · × S (t) D , (6)
Then the search space A(t) could be evolved to a better one A(t+1), with the update of S(t)i . Particularly, For a certain subspace S(t)i , we first get the subspace {α|α ∈ A(t) ∩αi = v (t) l } for each choice v (t) l ∈ S (t) i , where αi is the i
th dimension value of the architecture α. Then we compute the E-T Errors of the subspaces. We hypothesize there is a strong correlation between the continuous choices and the E-T Error in most cases, so we fit a linear function to estimate its tendency for simplification:
y = wx + b, (7)
where x ∈ R is the choices in S(t)i , and y ∈ R is the corresponding E-T Error. w and b are both scalar parameters. Then the new subspace S(t+1)i is given as:
S (t+1) i =
{ v
(t) j − ⌊w τ ⌋ γi ∣∣∣ v(t)j ∈ S(t)i }, (8) where τ is the threshold for search space evolving, γi is the steps for search dimension Si. Note that we remove the choices if they become less than zero. The detailed algorithm is also shown in Alg. 1.

heading: 2.4 Searching in the Searched Space

text:
Once the space search is done, we perform neural architecture search over the searched space. The search pipeline contains two sequential steps: 1) supernet training without resource constraints, and 2) evolution search under resource constraints.
The supernet training follows the same train recipe in space search. Similar to AutoFormer [6] and BigNas [52], the maximal, minimal subnets and several random architectures are sampled randomly from the evolved search space in each iteration and their corresponding weights are updated. We provide more details in Sec. 4.1. The evolution search process follows the similar setting in SPOS [12]. Our objective in this paper is to maximize the classification accuracy while minimizing the model size and FLOPs. Notably, vision transformer does not include any Batch Normalization [19],
hence the evolution process is much faster than CNN. We provide more details in the supplementary materials.

heading: 3 Analysis and Discussion

text:
In this section we provide analysis and discussion according to the space searching process. We hope that our observations can shed lights on both manual design of vision transformer architecture and the search space design for NAS methods. To minimize the prior knowledge used in search space design, we initialize the search space by setting a same search space for different stages. We provide the detailed search space and its evolution process in Fig. 2. Fig. 3 shows a direct illustration of space quality of the searched space focusing on the number of blocks.
The third stage is most important and increasing blocks in the third stage leads to better performance. As shown in the first row of Fig. 2, the blocks of stage 3 is constantly increasing from step 0 to step 2. We also observe that there are more blocks in stage 3 in some famous CNN networks, such as ResNet [14], EfficientNet [39], which share the same architecture characteristics with our finding in transformers. As a result, it seems that the third stage is the most important one. On the other hand, contrary to stage 3, the number of blocks decrease in stage 1,2,4 showing that these stages should include fewer layers.
Shallow layers should use a small window size while deep layers should use a larger one. As shown in the last row of Fig. 2, with the search steps increasing, the window size space decreases in stage 1 while increases in stage 3. It shows that shallow layers tend to have smaller window size while deep layers tend to have larger one. We assume that transformers increase focusing region with deeper layers. To verify this, we visualize the attention maps (top row in Fig. 4 (b,c) ) in different layers and
draw a box plot (bottom row in Fig. 4 (f,g)) to show the relative distances from the target position to its top-5/10 focusing positions. The attention maps show that the highest attentions (noted by numbers) expand with deeper layer, which is illustrated from (b) to (c). As for the average relative distance, deeper layers have larger medians and means, indicating larger focusing regions. The above observations are consistent with our assumption and provide solid evidence for it.
Deeper layer should use larger MLP ratio. Traditionally, all layers use the same MLP ratio. However, the third row of Fig. 2 illustrates that shallow layers should use small MLP ratio to result in better performance while deeper ones should use larger MLP ratio.
Q-K-V dimension should be smaller than the embedding dimension. In the original transformer block [41, 10], the Q-K-V dimension is the same as the embedding dimension. However, comparing the second row with the third row of Fig. 2, we find that the searched space has a slightly smaller Q-K-V dimension than embedding dimension, shown from stage 1 to stage 4. The gap between them are more obvious in the deeper layers. We conjecture the reason is that there are many heads with similar features in deeper layers. Therefore, the a relative small Q-K-V dimension is able to have great visual representation ability. We present more details in the supplementary materials.

heading: 4 Experiments

heading: 4.1 Implementation Details

text:
Space searching. The initial search space is presented in Fig. 2, the number of blocks and embedding dimension are set to {2, 3, 4} and {224, 256, 288} respectively for all stages, while the window size, number of heads, MLP ratio, Q-K-V dimensions are set to {7, 14}, {3, 3.5, 4}, {7, 8, 9}, {224, 256, 288} respectively for all blocks. We set the total iteration of space searching to 3, while setting the steps in Eq. (8) for block number, embed dim, MLP ratio, Q-K-V dim, windows size to 1, 64, 0.5, 64, 7, respectively. We set the threshold τ of space evolution in Eq. (8) as 0.4.
Supernet training. Similar to DeiT [40], we train the supernet with the following settings: AdamW [28] optimizer with weight decay 0.05, initial learning rate 1× 10−3 and minimal learning rate 1× 10−5 with cosine scheduler, 20 epochs warmup, batch size of 1024, 0.1 label smoothing, and stochastic depth with drop rate 0.2. We also adopt sandwich strategy as describe in Sec. 2.3. The models are trained for 300 epochs with 16 Nvidia Tesla 32G V100 GPUs.
Table 1: S3 performance on ImageNet in comparison with state-of-the-arts. We group the models according to their parameter sizes.
Models Top-1 Acc. Top-5 Acc. #Params. FLOPs Res. Model Type
ResNet50 [14] 76.2% 92.9% 25.5M 4.1G 2242 CNN RegNetY-4GF [35] 80.0% - 21.4M 4.0G 2242 CNN EfficietNet-B4 [39] 82.9% 95.7% 19.3M 4.2G 3802 CNN T2T-ViT-14 [53] 80.7% - 21.5M 6.1G 2242 Transformer DeiT-S [40] 79.9% 95.0% 22.1M 4.7G 2242 Transformer ViT-S/16 [10] 78.8% - 22.1M 4.7G 2242 Transformer Swin-T [27] 81.3% - 28.3M 4.5G 2242 Transformer S3-T (Ours) 82.1% 95.8% 28.1M 4.7G 2242 Transformer
ResNet152 [14] 78.3% 94.1% 60M 11G 2242 CNN EfficietNet-B7 [39] 84.3% 97.0% 66M 37G 6002 CNN BoTNet-S1-110 [38] 82.8% 96.4% 55M 10.9G 2242 CNN + Trans T2T-ViT-24 [53] 82.2% - 64M 15G 2242 Transformer Swin-S [27] 83.0% - 50M 8.7G 2242 Transformer S3-S (Ours) 83.7% 96.4% 50M 9.5G 2242 Transformer S3-S-384 (Ours) 84.5% 96.8% 50M 33G 3842 Transformer
RegNetY-16GF [35] 80.4% - 83M 16G 2242 CNN ViT-B/16 [10] 79.7% - 86M 18G 2242 Transformer DeiT-B [40] 81.8% 95.6% 86M 18G 2242 Transformer Swin-B [27] 83.3% - 88M 15.4G 2242 Transformer Swin-B-384 [27] 84.2% - 88M 47G 3842 Transformer S3-B (Ours) 84.0% 96.6% 71M 13.6G 2242 Transformer S3-B-384 (Ours) 84.7% 96.9% 71M 46G 3842 Transformer
79.6 79.8 80.0 80.2 80.4 80.6 80.8 81.0 81.2 Inherited Performance
79.6 79.8 80.0 80.2 80.4 80.6 80.8 81.0 81.2 Re tra in P er fo rm an ce
79.6 79.8 80.0 80.2 80.4 80.6 80.8 81.0 81.2 Inherited Performance
79.6 79.8 80.0 80.2 80.4 80.6 80.8 81.0 81.2 Fin et un e Pe rfo rm an ce
Figure 5: Comparison of subnets with inherited weights, fine-tuned and trained from scratch. 20 25 30 35 40 Error
0.0
0.2
0.4
0.6
0.8
1.0
Cu m
ula tiv
e pr
ob . AB
C Space A Space B Space C
Figure 6: Error empirical distribution function of different search spaces.

heading: 4.2 Ablation Study

text:
Effectiveness of once-for-all supernet sampling. To verify the effectiveness, we randomly sample 10 architectures from the well-trained supernet with the inherited weights, then compare the ImageNet classification performance of (1) models with only inherited weights, (2) models with finetuning, and (3) models retrained from scratch. As shown in Fig. 5, the architecture with inherited weights has comparable or even better performance when compared with the same architecture with retraining or finetuning. This phenomenon shows that our sampled architectures provide a precise and reliable performance proxy for space quality measurement.
Effectiveness of search space evolution. We conduct two experiments. In the first experiment, we randomly sample 1, 000 architectures with inherited weights from a search space, then compute and plot error empirical distribution, as in [35], of them. We denote the initial space, the one-step evolved space and the two-step evolved space by Space A, Space B and Space C respectively. As shown in Fig. 6, the space C has an obviously better overall quality compared to A and B. In the second experiment, we use evolution algorithm to search for top architectures in each space. Tab. 5 illustrates that the space C has a much better top architectures, being 16.6%/4.1% better than that in the spaces A and B. These results verify the effectiveness of search spaces evolution.

heading: 4.3 Image Classification

text:
We compare our searched models with state-of-the-arts on ImageNet [9]. As described in Sec. 2.1 and Sec. 4.1, we first evolve the search space, then optimize the weights of supernet and finally search on the supernet. For the experiments of high resolution images, the models are finetuned with 3842 inputs for 10 epochs, using an AdamW [28] optimizer with a constant learning rate of 10−5 and weight decay of 10−8.
We compare our S3 with state-of-the-art CNN models, vision transformers and hybrid CNNtransformer models. The results are reported in Tab. 1. Our proposed models S3-tiny/small/base (T/S/B for short) achieve 82.1%/83.7%/84.0% top-1 accuracy on ImageNet [9] using 2242 input images. Compared with vision transformer and hybrid models, our method obtains superior performance using similar FLOPs and parameters. For example, when considering parameters as the main constraint, we find a architecture that has 0.6% higher top-1 accuracy than Swin Transformer [27] while using 42% fewer parameters as shown in Fig. 7 . Compared to CNN models, our S3 models are also very competitive, especially for small parameter size. The proposed S3-S-384 (84.5%) performs slightly better than EfficientNet-B7 (84.3%) [39] while using 24% fewer parameters and 11% fewer FLOPs. For base models with fewer computational cost, S3-B also achieves superior performance to state-of-the-art methods, such as RegNet [35] and ViT [10].

heading: 4.4 Vision and Vision-Language Downstream Tasks

text:
To further evaluate the generalizability of the discovered architectures, we transfer them to vision and vision-language downstream tasks. We conduct experiments on COCO [24] for object detection, ADE20K [56] for semantic segmentation and VQA 2.0 [54] for visual question answering, respectively. The training and inference settings for object detection and semantic segmentation are the same as those in Swin transformer [27], while for visual language answering task, the settings are aligned with SOHO [18]. More experimental details are elaborated in supplementary materials.
Object Detection. As shown in Tab. 2, S3-T outperforms ResNet-50 [14] by +4.2 box AP and +3.7 mask AP under 3× learning schedule. Compared to DeiT-S [40], S3-T achieves +2.5 higher box AP and +2.4 higher mask AP. Increasing the model size and FLOPs, S3-S and S3-B surpass ResNeXt-101 [48] by +4.5 box AP, +3.9 mask AP and +4.2 box AP, +3.7 mask AP respectively. Compared to Swin-S and Swin-B, S3-S and S3-B outperform them by +0.8 box AP, +0.8 mask AP and +0.6 box AP, +0.4 mask AP respectively. All comparisons are conducted under the similar model
Table 3: Results on ADE20K Semantic Segmentation using UperNet. †: reproduced by the official code. ∗: without adding deconvolution in heads.
Method Backbone #param FLOPs mIoU mIoU(M) (G) (%) (ms+flip %) UperNet ResNet-50 [14] 67 951 42.05 42.78 UperNet DeiT-S∗ [40] 52 1099 43.15 43.85 UperNet Swin-T [27] 60 945 44.51 45.81 UperNet S3-T(Ours) 60 954 44.87 46.27 UperNet ResNet-101 [14] 86 1029 43.82 44.85 UperNet DeiT-B∗ [40] 121 2772 44.09 45.68 UperNet Swin-S† [27] 81 1038 47.60 48.87 UperNet S3-S(Ours) 81 1071 48.04 49.31
Table 4: Results on VQA v2.0 dataset.
Backbone #Param FLOPs VQA VQA(M) (G) test-dev(%) test-std(%)
ResNet-50 [14] 4.1 26 64.65 65.12 ResNet-101 [14] 7.8 45 65.44 65.71
Swin-T [27] 4.5 29 67.13 67.40 S3-T (Ours) 4.6 29 67.84 68.00
Table 5: Comparisons of searched architectures in different spaces.
Space #Param(M) FLOPs(G) Top-1(%) A 28M 4.7 65.5 B 29M 4.7 78.0 C 28M 4.7 82.1
size and FLOPs. Under the 1× training schedule, our S3 model is also superior to other backbones, demonstrating its robustness and good transferability.
Semantic Segmentation. We report the mIoU score on ADE20K [56] validation set in Tab. 3. It can be seen that our S3 models consistently achieve better performance than ResNet [14], DeiT [40] and Swin [27]. Specifically, S3-T is +3.5 mIoU superior to ResNet-50. Compared to other transformerbased models, S3-T is +2.42 mIoU higher than DeiT-S, and +0.46 mIoU higher than Swin-T. S3-S outperforms ResNet-101 with a large margin (+4.46). It also surpasses DeiT-B and Swin-S by +3.63 and +0.44 mIoU, respectively. Note that we tried to add deconvolution layers to build a hierarchical structure for DeiT, but did not get performance improvements.
Visual Question Answering. As shown in Tab. 4, we report the accuracy on the test-dev and test-std set of VQA v2.0 dataset [54] for different vision backbones. Our model S3-T brings more than +2% accuracy gains over ResNet-101 [14], and surpassing Swin-T [27] by around 0.6% accuracy under aligned model complexity. All the above experiments demonstrate the good generality of the searched models as well as proving the effectiveness of the searched design space of vision transformer.

heading: 5 Related Work

text:
Vision Transformer. Originally designed for natural language processing, Transformer has been recently applied to vision tasks [4, 55, 5, 46]. ViT [10] is the first one that introduces a pure transformer architecture for visual recognition. Alleviating the data hungry and computational issues, DeiT [40] introduces a series of training strategies to remedy the limitations. Some further works resort to feature pyramids to purify multi-scale features and model global information [44, 15, 31]. Swin Transformer [27] introduces non-overlapping window partitions and restricts the calculation of self-attention within each window, leading to a linear runtime complexity w.r.t. token numbers.
Search Space. There is a wide consensus that a good search space is crucial for NAS. Regarding CNN, the current search space, such as cell-based space [26, 7, 49] and MBConv-based space [3, 32, 51], is pre-defined and fixed during search. Several recent works [17, 29, 30, 33, 25, 23] perform search space shrinking to find a better compact one. Moreover, NSENet [8] proposes an evolving search space algorithm for MBConv space optimization. RegNet [35] presents a new paradigm for understanding design space and discovering design principles. However, the exploration of vision transformer space is still limited, and our work is the first along the direction.
Search Algorithm. There has been an increasing interest in NAS for automating network design [11, 20]. Early approaches use either reinforcement learning [57, 58] or evolution algorithms [47, 36]. These approaches require training thousands of architecture candidates from scratch. Most recent works resort to the one-shot weight sharing strategy to amortize the searching cost [22, 34, 1, 12]. The key idea is to train an over-parameterized supernet where all subnets share weights. For transformer, there are few works employing NAS to improve architectures. Evolved Transformer [37] searches the inner architecture of transformer block. HAT [43] proposes to design hardware-aware transformers with NAS to enable low-latency inference on resource-constrained hardware platforms. BossNAS [21] explores hybrid CNN-Transformers with block-wisely self-supervised. The most related work is AutoFormer [6] which provides a simple yet effective framework of vision transformer search. It could achieve once-for-all training with newly proposed weight entanglement. In this work, we focus on both the search space design and architectures of vision transformer.

heading: 6 Conclusion

text:
In this work, we propose to search the search space of vision transformer. The central idea is to gradually evolve different search dimensions guided by their E-T Error computed using a weightsharing supernet. We also provide analysis on vision transformer, which might be helpful for understanding and designing transformer architectures. The search models, S3, achieve superior performance to recent prevalent ViT and Swin transformer model families under aligned settings. We further show their robustness and generality on downstream tasks. In further work, we hope to investigate the usage of S3 in CNN search space design.
Acknowledgements and Disclosure of Funding
Thank Hongwei Xue for providing helps and discussions on vision-language tasks. No external funding was received for this work.

title: Searching the Search Space of Vision Transformer