summaries: The paper solves a problem similar to Neural Body (reconstruction / free-view rendering of clothed humans from videos based on SMPL fits) but does not require training a neural network per scene; instead, it generalises from training sequences and, at test time, runs only a feed-forward model conditioned on keyframes. This is important for many applications since training a neural net per scene is not practical, and also the proposed extension is not trivial: the paper brings novel ideas (using attention for time-wise and camera-wise aggregation). Experimental evaluation is comprehensive and the quality is even superior to per-scene optimised Neural Bodies. Given this, I suggest to accept the paper, although there is some room for improvement. This paper presents a method to synthesize a novel view image of a person given the sparse multiview images. The paper proposes time-augmented skeletal features---the pixel-aligned features that encode NeRF-like rendering fields by aggregating image features across  time and views, parametrized by a human mesh model. They use transformer networks to learn fusing multiview image features over time. The resulting features are used to generate an implicit function that can predict RGB and opacity. They demonstrate that the proposed method is generalizable to unseen identities and poses, validated on ZJU-MoCap and AIST datasets.   This work proposes a method that can synthesize free-viewpoint videos for dynamic humans in a sparse multi-camera system.  As is claimed and demonstrated in the paper, by introducing transformer modules to aggregate spatio-temporal information in the sparse multi-view data, the learned neural radiance fields are generalizable to unseen human identities and unseen poses.  The proposed method is tested on public multi-camera datasets, showing promising results. This paper presents a generalizable NeRF network that can produce novel view synthesis for novel dynamic human scenes from sparse camera views, in a feed-forward manner. It leverages visual features from tracked parametric body model and similar to PixelNeRF, it conditions the NeRF representation on pixel-aligned image features for appearance generalization. A novel combination of a temporal Transformer and a multiview Transformer is proposed to integrate multi-frame and multi-view visual observations. It demonstrated stronger generalization capability than prior work.  
abstractText: In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, is highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture. Specifically, we first introduce a temporal transformer that aggregates tracked visual features based on the skeletal body motion over time. Moreover, a multi-view transformer is proposed to perform cross-attention between the temporally-fused features and the pixel-aligned features at each time step to integrate observations on the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses. The video results and code are available at https://youngjoongunc.github.io/nhp.
authors.name: Youngjoong Kwon
authors.name: Dahun Kim
authors.name: Duygu Ceylan
authors.name: Henry Fuchs
references.authors: Kara-Ali Aliev Dmitry Ulyanov Victor Lempitsky
references.title: Neural point-based graphics
references.venue: arXiv preprint arXiv:1906.08240,
references.authors: Nicolas Carion Francisco Massa Gabriel Synnaeve Nicolas Usunier Alexander Kirillov Sergey Zagoruyko
references.title: End-to-end object detection with transformers
references.venue: In European Conference on Computer Vision,
references.authors: Joel Carranza Christian Theobalt Marcus A Magnor Hans-Peter Seidel
references.title: Free-viewpoint video of human actors
references.venue: ACM transactions on graphics (TOG),
references.authors: Alvaro Collet Ming Chuang Pat Sweeney Don Gillett Dennis Evseev David Calabrese Hugues Hoppe Adam Kirk Steve Sullivan
references.title: High-quality streamable free-viewpoint video
references.venue: ACM Transactions on Graphics (ToG),
references.authors: Edilson De Aguiar Carsten Stoll Christian Theobalt Naveed Ahmed Hans-Peter Seidel Sebastian Thrun
references.title: Performance capture from sparse multi-view video
references.venue: In ACM SIGGRAPH
references.authors: Paul Debevec Tim Hawkins Chris Tchou Haarm-Pieter Duiker Westley Sarokin Mark Sagar
references.title: Acquiring the reflectance field of a human face
references.venue: In Proceedings of the 27th annual conference on Computer graphics and interactive techniques,
references.authors: Mingsong Dou Sameh Khamis Yury Degtyarev Philip Davidson Sean Ryan Fanello Adarsh Kowdle Sergio Orts Escolano Christoph Rhemann David Kim Jonathan Taylor
references.title: Fusion4d: Real-time performance capture of challenging scenes
references.venue: ACM Transactions on Graphics (TOG),
references.authors: John Flynn Michael Broxton Paul Debevec Matthew DuVall Graham Fyffe Ryan Overbeck Noah Snavely Richard Tucker
references.title: Deepview: View synthesis with learned gradient descent
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Guy Gafni Justus Thies Michael Zollhofer Matthias Nießner
references.title: Dynamic neural radiance fields for monocular 4d facial avatar reconstruction
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Juergen Gall Carsten Stoll Edilson De Aguiar Christian Theobalt Bodo Rosenhahn Hans-Peter Seidel
references.title: Motion capture using joint skeleton tracking and surface estimation
references.venue: IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Chen Gao Yichang Shih Wei-Sheng Lai Chia-Kai Liang Jia-Bin Huang
references.title: Portrait neural radiance fields from a single image
references.venue: arXiv preprint arXiv:2012.05903,
references.authors: Ke Gong Xiaodan Liang Yicheng Li Yimin Chen Ming Yang Liang Lin
references.title: Instance-level human parsing via part grouping network
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Kaiwen Guo Peter Lincoln Philip Davidson Jay Busch Xueming Yu Matt Whalen Geoff Harvey Sergio Orts-Escolano Rohit Pandey Jason Dourgarian
references.title: The relightables: Volumetric performance capture of humans with realistic relighting
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Youngjoong Kwon Stefano Petrangeli Dahun Kim Haoliang Wang Henry Fuchs Viswanathan Swaminathan
references.title: Rotationally-consistent novel view synthesis for humans
references.venue: In Proceedings of the 28th ACM International Conference on Multimedia,
references.authors: Youngjoong Kwon Stefano Petrangeli Dahun Kim Haoliang Wang Eunbyung Park Viswanathan Swaminathan Henry Fuchs
references.title: Rotationally-temporally consistent novel view synthesis of human performance video
references.venue: In European Conference on Computer Vision,
references.authors: Ruilong Li Shan Yang David A Ross Angjoo Kanazawa
references.title: Ai choreographer: Music conditioned 3d dance generation with aist++
references.venue: In International Conference on Computer Vision,
references.authors: Tianye Li Mira Slavcheva Michael Zollhoefer Simon Green Christoph Lassner Changil Kim Tanner Schmidt Steven Lovegrove Michael Goesele Zhaoyang Lv
references.title: Neural 3d video synthesis
references.venue: arXiv preprint arXiv:2103.02597,
references.authors: Yiyi Liao Katja Schwarz Lars Mescheder Andreas Geiger
references.title: Towards unsupervised learning of generative models for 3d controllable image synthesis
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Baoyuan Liu Min Wang Hassan Foroosh Marshall Tappen Marianna Pensky
references.title: Sparse convolutional neural networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Lingjie Liu Jiatao Gu Kyaw Zaw Lin Tat-Seng Chua Christian Theobalt
references.title: Neural sparse voxel fields
references.venue: arXiv preprint arXiv:2007.11571,
references.authors: Lingjie Liu Weipeng Xu Michael Zollhoefer Hyeongwoo Kim Florian Bernard Marc Habermann Wenping Wang Christian Theobalt
references.title: Neural rendering and reenactment of human actor videos
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Shaohui Liu Yinda Zhang Songyou Peng Boxin Shi Marc Pollefeys Zhaopeng Cui
references.title: Dist: Rendering deep implicit signed distance function with differentiable sphere tracing
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Stephen Lombardi Tomas Simon Jason Saragih Gabriel Schwartz Andreas Lehrmann Yaser Sheikh
references.title: Neural volumes: Learning dynamic renderable volumes from images
references.authors: Matthew Loper Naureen Mahmood Javier Romero Gerard Pons-Moll Michael J Black
references.title: Smpl: A skinned multi-person linear model
references.venue: ACM transactions on graphics (TOG),
references.authors: Lars Mescheder Michael Oechsle Michael Niemeyer Sebastian Nowozin Andreas Geiger
references.title: Occupancy networks: Learning 3d reconstruction in function space
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Ben Mildenhall Pratul P Srinivasan Matthew Tancik Jonathan T Barron Ravi Ramamoorthi Ren Ng
references.title: Nerf: Representing scenes as neural radiance fields for view synthesis
references.venue: In European Conference on Computer Vision,
references.authors: Ryota Natsume Shunsuke Saito Zeng Huang Weikai Chen Chongyang Ma Hao Li Shigeo Morishima
references.title: Siclope: Silhouette-based clothed people
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Michael Niemeyer Lars Mescheder Michael Oechsle Andreas Geiger
references.title: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Kyle Olszewski Sergey Tulyakov Oliver Woodford Hao Li Linjie Luo
references.title: Transformable bottleneck networks
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Keunhong Park Utkarsh Sinha Jonathan T Barron Sofien Bouaziz Dan B Goldman Steven M Seitz Ricardo-Martin Brualla
references.title: Deformable neural radiance fields
references.venue: arXiv preprint arXiv:2011.12948,
references.authors: Keunhong Park Utkarsh Sinha Peter Hedman Jonathan T Barron Sofien Bouaziz Dan B Goldman Ricardo Martin-Brualla Steven M Seitz
references.title: Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields
references.venue: arXiv preprint arXiv:2106.13228,
references.authors: Sida Peng Junting Dong Qianqian Wang Shangzhan Zhang Qing Shuai Xiaowei Zhou Hujun Bao
references.title: Animatable neural radiance fields for modeling dynamic human bodies
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Sida Peng Yuanqing Zhang Yinghao Xu Qianqian Wang Qing Shuai Hujun Bao Xiaowei Zhou
references.title: Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans
references.venue: In CVPR,
references.authors: Songyou Peng Michael Niemeyer Lars Mescheder Marc Pollefeys Andreas Geiger
references.title: Convolutional occupancy networks
references.venue: arXiv preprint arXiv:2003.04618,
references.authors: Albert Pumarola Enric Corona Gerard Pons-Moll Francesc Moreno-Noguer
references.title: D-nerf: Neural radiance fields for dynamic scenes
references.venue: arXiv preprint arXiv:2011.13961,
references.authors: Amit Raj Michael Zollhoefer Tomas Simon Jason Saragih Shunsuke Saito James Hays Stephen Lombardi
references.title: Pva: Pixel-aligned volumetric avatars
references.venue: arXiv preprint arXiv:2101.02697,
references.authors: Shunsuke Saito Zeng Huang Ryota Natsume Shigeo Morishima Angjoo Kanazawa Hao Li
references.title: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Shunsuke Saito Tomas Simon Jason Saragih Hanbyul Joo
references.title: Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Shaoshuai Shi Chaoxu Guo Li Jiang Zhe Wang Jianping Shi Xiaogang Wang Hongsheng Li
references.title: Pv-rcnn: Point-voxel feature set abstraction for 3d object detection
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Vincent Sitzmann Justus Thies Felix Heide Matthias Nießner Gordon Wetzstein Michael Zollhofer
references.title: Deepvoxels: Learning persistent 3d feature embeddings
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Vincent Sitzmann Michael Zollhöfer Gordon Wetzstein
references.title: Scene representation networks: Continuous 3d-structure-aware neural scene representations
references.venue: arXiv preprint arXiv:1906.01618,
references.authors: Carsten Stoll Juergen Gall Edilson De Aguiar Sebastian Thrun Christian Theobalt
references.title: Video-based reconstruction of animatable human characters
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Zhuo Su Lan Xu Zerong Zheng Tao Yu Yebin Liu
references.title: Robustfusion: Human volumetric capture with data-driven visual cues using a rgbd
references.authors: Justus Thies Michael Zollhöfer Matthias Nießner
references.title: Deferred neural rendering: Image synthesis using neural textures
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Shuhei Tsuchida Satoru Fukayama Masahiro Hamasaki Masataka Goto
references.title: Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing
references.venue: In ISMIR,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: arXiv preprint arXiv:1706.03762,
references.authors: Qianqian Wang Zhicheng Wang Kyle Genova Pratul Srinivasan Howard Zhou Jonathan T. Barron Ricardo Martin-Brualla Noah Snavely Thomas Funkhouser
references.title: Ibrnet: Learning multi-view image-based rendering
references.authors: Xiaolong Wang Ross Girshick Abhinav Gupta Kaiming He
references.title: Non-local neural networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Minye Wu Yuehao Wang Qiang Hu Jingyi Yu
references.title: Multi-view neural human rendering
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Wenqi Xian Jia-Bin Huang Johannes Kopf Changil Kim
references.title: Space-time neural irradiance fields for free-viewpoint video
references.venue: arXiv preprint arXiv:2011.12950,
references.authors: Alex Yu Vickie Ye Matthew Tancik Angjoo Kanazawa
references.title: pixelnerf: Neural radiance fields from one or few images
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Wentao Yuan Zhaoyang Lv Tanner Schmidt Steven Lovegrove
references.title: Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering
references.venue: arXiv preprint arXiv:2101.01602,
references.authors: Zerong Zheng Tao Yu Yixuan Wei Qionghai Dai Yebin Liu
references.title: Deephuman: 3d human reconstruction from a single image
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Tinghui Zhou Richard Tucker John Flynn Graham Fyffe Noah Snavely
references.title: Stereo magnification: Learning view synthesis using multiplane images
references.venue: arXiv preprint arXiv:1805.09817,
sections.heading: 1 Introduction
sections.text: Free-viewpoint video of a human performer has a variety of applications in the area of telepresence, mixed reality, gaming and etc. Conventional free-viewpoint video systems require extremely expensive setups such as dense camera rigs [4, 7, 43] or accurate depth sensors [6, 13], to capture person-specific appearance information. In this paper, we aim at a scalable solution for free-viewpoint human performance rendering that generalizes across different human performers and requires only sparse camera views. However, representing and rendering arbitrary human performances is extremely challenging when the observations are highly sparse (up to three to four views) due to heavy self-occlusions and dynamic articulations of the body parts. In particular, an effective solution needs to coherently aggregate appearance information from sparse multi-view observations across time as the body undergoes a 3D motion. Furthermore, the solution needs to generalize to unseen motions and characters at test time.
  Recently, neural radiance fields (NeRF) [26, 11, 17, 30, 33, 35, 36, 47, 50, 52, 53] have shown photo-realistic novel view synthesis results in per-scene optimization settings. To avoid the expensive per-scene training and improve the practicality, generalizable NeRFs [36, 52, 47] have been proposed which use image-conditioned, pixel-aligned features and achieve feed-forward view synthesis from sparse input views [36, 52]. Direct application of these methods to complex and non-rigid human motion is not straightforward, however, and naive solutions suffer from significant artifacts as shown in Fig. 3. Some existing methods [37, 52] aggregate image features across multiple views by simple average pooling, which often leads to over-smoothed outputs when details observed from multiple
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  views (e.g., front and side views) differ due to self occlusions of humans. Alternatively, several methods [23, 33] have proposed to learn person-specific global appearance features from multi-view observations. However, such methods are not able to generalize to new human performers.
  To address these challenges, we propose Neural Human Performer, a novel approach that learns generalizable radiance fields based on a parametric 3D body model for robust performance capture. In addition to exploiting a parametric body model as a geometric prior, the core of our method is a combination of temporal and multi-view transformers which help to effectively aggregate spatio-temporal observations to robustly compute the density and color of a query point. First, the temporal transformer aggregates trackable visual features based on the input skeletal body motion across time. The following multi-view transformer performs cross-attention between the temporallyaugmented skeletal features and the pixel-aligned features for each time step. The proposed modules collectively contribute to the adaptive aggregation of multi-time and multi-view information, resulting in significant improvements in synthesis results in different generalization settings of unseen motions and identities.
  We study the efficacy of Neural Human Performer on two multi-view human performance capture datasets, ZJU-MoCap [33] and AIST [16]. Experiments show that our method significantly outperforms recent generalizable radience field (NeRF) methods. Furthermore, we compare ours with identity-specific methods [33, 44, 49] that also utilize a 3D human body model prior. Surprisingly, our generalized method achieves better rendering quality than the person-specific dedicated methods when tested on novel poses demonstrating the effectiveness of our transformer-based generalizable representation.
  To summarize, our contributions are:
  • We present a new feed-forward method of synthesizing novel-view videos of arbitrary human performers from sparse camera views. We propose Neural Human Performer that learns generalizable neural radiance representations by leveraging a 3D body motion prior.
  • We design a combination of temporal and multi-view transformers that can aggregate information on the fly over video frames across multiple views to render each frame of the novel-view video.
  • We show significant improvements over recent generalizable NeRF methods on unseen identities and poses. Moreover, our generalization results can outperform even personspecific methods when tested on unseen poses.
sections.heading: 2 Related works
sections.text: Human performance capture. Novel view synthesis of human performance has a long history in computer vision and graphics. Traditional methods rely on complicated hardware such as dense camera rigs [4, 7, 43] or accurate depth sensors [6, 13]. To enable free-view video from sparse camera views, template-based methods [3, 5, 10, 42] exploit pre-scanned human models to track the motion of a person. However, their synthesis results are not photo-realistic and pre-scanned human models are not available in most cases. Recent methods [27, 37, 38, 54] learn 3D human geometry priors along with pixel aligned features to enable detailed 3D human reconstructions even from single images. However, these methods often suffer under complex human poses that are never seen during training and hence cannot be directly used for our purpose of human performance synthesis.
  Neural 3D representations. Recently, there has been great progress in learning neural networks to represent the shape and appearance of scenes. The 3D representations are learned from 2D images via differentiable rendering networks. Convolutional neural networks are used to predict volumetric representations via 3D voxel-grid features [40, 23, 29, 25, 14, 15], point clouds [1, 49], textured meshes [18, 21, 44] and multi-plane images [8, 55]. The learnt representations are projected by a 3D-to-2D operation to synthesize images. Some methods [14, 15] introduce consistency loss to improve the multi-view and temporal consistency. However, these methods often have difficulty in scaling to higher resolution due to memory restrictions.
  To eschew these problems, implicit function-based methods [20, 22, 28, 41] learn a multi-layer perceptron that directly translates a 3D positional feature into a pixel generator. The more recent NeRF [26] learns implicit fields of density and color with a volume rendering technique and achieves photo-realistic view synthesis. Among many following NeRF extensions, [30, 31, 35, 53, 17, 50]
  focus on dynamic scenes. While showing impressive results, it’s an extremely under-constrained problem to jointly learn NeRF and highly dynamic deformation fields. To regularize the training, Neural Body [33] combines NeRF with a deformable human body model (e.g., SMPL [24]). Despite the promising results, these general deformable NeRF [17, 53] and human-specific NeRF [11, 9, 33, 32] methods must be optimized for each new video separately, and generalize poorly on unseen scenarios. Generalizable NeRFs [36, 47, 52] try to avoid the expensive per-scene optimization by image-conditioning using pixel-aligned features. However, directly extending such methods to model complex and dynamic 3D humans is not straightforward when available observations are highly sparse. Unlike existing works, our method exploits temporal and multi-view information on-the-fly and achieves free-viewpoint human rendering in a feed-forward manner, also generalizing to new, unseen human identities and poses.
sections.heading: 3 Method
sections.heading: 3.1 Neural Human Performer
sections.text: Problem definition. In our setting, given a sparse set (e.g., 3 or 4) of multi-view cameras c = 1, . . . , C, input videos of an arbitrary human performance Ic,1:T := {Ic,1, Ic,2, ..., Ic,T } are captured for each camera view c defined by {Kc, [R|t]c}. We assume that the 3D human body model fit corresponding to each frame is given. Our goal is to synthesize a novel view video Îq,1:T for a query viewpoint q defined by {Kq, [R|t]q}.
  Overview. To compensate for the sparsity of available input views, we propose to exploit temporal information across video frames. In practice, we sample M memory frames from the original input videos to augment each queried timestep t. Our goal is to learn generalizable 3D representations of human performers from multi-time (M ) and multi-view (C) observations.
  To this end, the Neural Human Performer is proposed with two main components as illustrated in Fig. 1. First, we construct the time-augmented skeletal features {s}. We exploit a human body model (SMPL [24]) to construct 3D skeletal features by projecting all the SMPL vertices onto each memory frame and picking up the pixel-aligned image features [36, 37, 52] at the projected 2D locations. The skeletal features are sampled from all memory frames to construct the skeletal feature bank. Inspired by Transformers[2, 46, 48], we propose a temporal Transformer that aggregates these memory features into the time-augmented skeletal features {s′}. In the second stage, given a query 3D point x, skeletal features are sampled at x. In addition, pixel-aligned features {p} at each time t are sampled by directly projecting x onto the input images {I}t. The multi-view Transformer is proposed to learn the correlation between the pixel-aligned features {p} and the time-augmented skeletal features {s′}, and to adaptively fuse the multi-view
  information. The final fused feature of the query point x is fed into the radiance field module to predict its color and density.
sections.heading: 3.2 Construction of time-augmented skeletal features
sections.text: Unlike static scenes, video inputs of moving characters inherently contain more visual cues as the occluded regions in a frame may be visible in other (potentially distant) frames. To take advantage of this temporal information, we first build up the skeletal feature bank from memory frames by leveraging a parametric body model (see Fig. 1). Then, we propose a temporal Transformer module that aggregates such collected features.
  For each view c and time t, we first build frame-level skeletal features sc,t ∈ RL×d by sampling image features at the projection of the SMPL vertices in each image Ic,t ∈ RH×W×3. L denotes the number of SMPL vertices and d is the dimension of image features.
  After collecting all the skeletal features from all memory frames, we aggregate the time information in an attention-aware manner, instead of using simple average pooling. Specifically, for any ith skeletal feature vertex sic,t ∈ Rd, the proposed temporal Transformer casts attention over all other features contained in the vertex’s memory bank sic,t1:tM = {s i c,t1 , s i c,t2 , ..., s i c,tM } ∈ R
  M×d. In particular, we compute soft weights for all memory feature vertices in a non-local manner with respect to the current timestep t. Then, the values of the memory features are weighted summed as
  t_attic,t = ψ ( 1√
  d0 q(sic,t) · k(sic,t1:tM )
  T ) , t_attic,t ∈ R1×M
  s′c,t i = t_attic,t · v(sic,t1:tM ) + s i c,t, s ′ c,t ∈ RL×d ∀i
  (1)
  where ψ represents the softmax operator along the second axis, q(·), k(·) and v(·) are learnable query, key and value embedding functions Rd→d0 of the temporal Transformer. In other words, the representation sic,t of each skeletal vertex at time t is computed through a dynamically weighted combination of all its previous and next representations in the memory frames. This allows our network to incorporate helpful information and ignore irrelevant ones from other timesteps. In practice, the temporal Transformer operation in Eq. (1) is performed by a batch matrix multiplication for all skeletal vertices L and all available viewpoints C.
sections.heading: 3.3 Multi-view aggregation of skeletal and query features
sections.text: Given a query 3D point x ∈ R3, we retrieve the corresponding (time-augmented) skeletal feature s′c,t
  x ∈ Rd at the queried location via trilinear interpolation in the SMPL space with SparseConvNet [19], following [39, 34, 51, 33].
  In addition, we sample pixel-aligned image feature pxc,t via direct R3→2 projection of the query point x on Ic,t. It is important to note that the pixel-aligned feature pxc,t is time-specific and represents the exact query location of x, while the skeletal feature s′c,t
  x is time-augmented (w.r.t t) and contains inherent geometric deviations in the SMPL vertices and the following trilinear interpolations. We propose to combine these two complementary features, which will be shown to be effective in Sec. 4.4.
  Given the two sets of multi-view features, skeletal s′1:C,t x = {s′c,t x|c = 1, ..., C} ∈ RC×d and pixel-aligned px1:C,t = {pxc,t|c = 1, ..., C} ∈ RC×d, we propose a multi-view Transformer that performs cross-attention from skeletal to pixel-aligned features. Specifically, the values of pixelaligned features from all viewpoints is re-weighted based on how much compatible they are with each skeletal features. The non-local cross-attention mv_att is constructed as:
  mv_attxt = ψ ( 1√
  d1 k(s′1:C,t x ) · k(px1:C,t)T
  ) , mv_attxt ∈ RC×C
  zx1:C,t = mv_att x t · v(px1:C,t) + v(s′1:C,t x ), zx1:C,t ∈ RC×d, zxc,t ∈ R1×d
  (2)
  where ψ represents the softmax operator along the second axis. Note that k and v are new layers independent from those in the temporal Transformer. The confident observations in each view will have large weights and be highlighted, and vice versa. Finally, we use the view-wise mean of zxt = 1 C ∑ c z x c,t ∈ Rd as our meta-time and meta-view representation of the query point x.
  The final density σt(x) and color values rgbt(x) at time t are computed as:
  σt(x) =MLPσ(z x t ), rgbt(X) =MLPrgb( ∑ c (zxc,t; γd(d))/C), (3)
  where MLPσ and MLPrgb consist of four and two linear layers respectively, and γd : R3→6×l is a positional encoding of viewing direction d ∈ R3 as in [26] with 2× l different basis functions. More details on the network architecture can be found in the supplementary material.
sections.heading: 3.4 Volume Rendering
sections.text: The predicted color of a pixel p ∈ R2 for a target viewpoint q in the focal plane of the camera and center r0 ∈ R3 is obtained by marching rays into the scene using the camera-to-world projection matrix, P−1 = [Rq|tq]−1K−1q with the direction of the rays given by d = P−1p−r0 ∥P−1p−r0∥ .
  We then accumulate the radiance and opacity along the ray r(z) = r0 + zd for z ∈ [znear, zfar] as defined in NeRF [26] as follows:
  Iq(p) = ∫ zfar znear T(z)σ(r(z))c(r(z),d)dz, where T(z) = exp ( − ∫ z znear σ(r(s))ds ) (4)
  In practice, we uniformly sample a set of 64 points z ∼ [znear, zfar]. We set X = r(z) and use the quadrature rule to approximate the integral. We compute the 3D bounding box of the SMPL parameters at time t and derive the bounds for ray sampling znear, zfar.
sections.heading: 3.5 Loss Function
sections.text: Given the ground truth target images Iq,t, we train both the radiance field and feature extraction networks using a simple photometric reconstruction loss L = ∥Îq,t − Iq,t∥2 .
sections.heading: 4 Experiments
sections.text: We present novel view synthesis and 3d reconstruction results of human performances in different generalization scenarios. We compare our method against the current best view-synthesis methods from two classes: per-subject optimization methods that also use a human body model prior (Sec. 4.1) and generalizable NeRF methods (Sec. 4.2). We experiment on the ZJU-MoCap [33] and AIST datasets [45, 16]. For training and testing our model as well as the baselines, we remove the background using the foreground masks that are either provided by the dataset or pre-computed using an off-the-shelf method. Unless otherwise specified, we sample two memory frames {t− 20, t+ 20} at time t (i.e., three timesteps in total) and take three canonical input views in all experiments. The details of the implementation, datasets, training process are provided in the supplementary material.
sections.heading: 4.1 Comparison with body model-based, per-subject optimization methods.
sections.text: Baselines. We perform comparisons with the state-of-the-art Neural Body (NB) [33] that combines the body model prior SMPL and NeRF in a per-subject optimization setting. Neural Textures (NT) [44] renders a coarse mesh with latent texture maps and uses a 2D CNN to render target images. We use the SMPL mesh as the coarse mesh input to NT in our experiments. NHR [49] extracts 3D features from input point clouds and renders them into 2D images. Since dense point clouds are difficult to obtain from sparse camera views, we use the SMPL vertices as input point clouds. These methods have reported that they can adapt to new poses of the same performer, hence we compare to them for the task of novel pose synthesis.
  Setup. We experiment with ZJU-MoCap dataset [33] which provides performance captures of 10 human subjects captured from 23 synchronized cameras, human body model parameters as well as the foreground mask corresponding to each frame. Each video contains complex motions such as kicking and Taichi and is between 1000 to 2000 frames long. We first split the dataset into two parts: source and target videos. In all comparisons, the first 300 frames of either source or target videos are used during training, and the remaining next frames (unseen poses) are held out for testing. Note that
  Method PSNR SSIM Trained on source subjects NB 23.79 0.887 NHR 22.31 0.871 NT 22.28 0.872 Trained on source subjects Ours 26.94 0.929
  a . Test results on unseen poses of source subjects
  Method PSNR SSIM Trained on target subjects NB 22.88 0.880 NHR 22.03 0.875 NT 21.92 0.873 Trained on source subjects Ours 24.75 0.906
  b . Test results on unseen poses of target subjects
  Method PSNR SSIM Trained on source subjects NB 28.51 0.947 NHR 23.95 0.897 NT 23.86 0.896 Trained on source subjects Ours 28.73 0.936 Ours per-subject 31.57 0.966
  c . Test results on seen poses of source subjects
  Table 1: Comparison with other body model-based, per-subject optimization methods.
  Figure 2: Pose generalization – comparison with per-scene optimization methods that use a human body prior. Results of NT(Neural textures) [44], NHR (Neural human rendering) [49], NB (Neural body) [33] and ours. Novel view synthesis on the ZJU-MoCap for unseen poses of source subjects. While ours is a single model trained on all source subjects; baselines are trained in a per-subject manner.
  the baseline methods are always trained in a per-subject manner. To validate whether the training is reproducible, we experiment with 5 independent runs with random train/test splits and observe a variance of 0.15 PSNR, showing that the results are consistent. In each independent run, we used seven models for training and the other three for testing.
  Results. We present three different comparison settings to validate our method. We would like to point out that all the comparison settings place our method (‘ours’) in disadvantage. This is because our model is trained on all the source subjects at once (one network for all subjects), while the competing methods are per-subject trained on the subject to be tested (one network for one subject) - easier setting. First, we evaluate the pose generalization capability (Table. 1a and Fig. 2) where we train on the training splits of the source subjects and test on the testing splits, i.e., unseen poses, of the same source subjects. Our method significantly outperforms all the baselines and the state-of-the-art Neural Body [33] by +3.15 PSNR and +0.042 SSIM scores. We next evaluate the identity generalization capability (Table. 1b) by testing on the unseen poses of target subjects. While our method is trained on source subjects only, baselines are trained on target subjects. Note that this comparison is disadvantageous to us since unlike our method the baselines have seen the testing subjects as they must be trained separately for each subject. Surprisingly, however, our method still outperforms all the baselines by a healthy margin of +1.87 PSNR and +0.026 SSIM scores. These results indicate that our proposed architecture with the temporal and the multi-view transformers generalizes well across unseen identities and poses, and produces photo-realistic results. Finally, in Table. 1c, we evaluate our method and baselines on seen poses of seen subjects. Even when we train a single model of our method for all the source sobjects (‘ours’), ours achieves comparable results with the state-of-the-art per-subject based methods [33]. To make a more fair comparison, we also train our method in the same per-subject setting similar to the baselines. Table. 1c shows that in this case, our model (‘ours per-subject’) achieves a significant improvement over the best-performing baseline [33] by +3 PSNR and +1.4% SSIM.
sections.heading: 4.2 Comparison with generalizable NeRF methods.
sections.text: Baselines. Among the recent generalizable NeRF methods [36, 52, 47], we compare with PixelNeRF [52] and PVA [36] which focus on very sparse (up to 3 or 4) input views. We reimplement [36] since the code is not available.
  Setup. In addition to the ZJU-MoCap dataset (see Sec. 4.1), we experiment on the larger AIST dataset [45, 16] to further evaluate the generalization abilities of different methods. AIST dataset provides dance videos of 30 human subjects captured from 9 cameras, together with the corresponding SMPL fits. We extract the foregroud mask of each image using an off-the-shelf human parser [12]. AIST dataset contains highly diverse motions, slow to fast, simple to complex. We split the dataset into 20 and 10 subjects for training and testing respectively, where the testing dataset contains unseen poses of unseen subjects.
  Novel view synthesis results. As shown in Table. 2, across all the datasets and all the metrics, our method consistently outperforms the baselines by healthy margins of +1.6 PSNR and +0.037 SSIM scores. Fig. 3 and Fig. 5 present the same tendency in visualizations. Pixel-NeRF and PVA aggregate multi-view observations via average pooling without explicitly considering the correlation between the views. In contrast, our temporal and multi-view transformers learn to model the correlation between input views and integrate different observations to help the NeRF module to produce higher quality results. Another advantage of our method is that the used body model prior provides a robust geometric cue to handle the self-occlusion of the human body.
  3D reconstruction results. We also evaluate the 3D reconstruction quality of generalizable NeRF methods and our method on the ZJU-MoCap (Fig. 4) and AIST datasets (Fig. 5) given three input views. The visualizations show that our 3D reconstructions align well with the input images, and is more accurate than even the per-subject method [33] (e.g., the shape of the upper cloth in Fig. 4).
  Overall, these results verify that since humans are complex and prone to occlusions, more sophisticated designs compared to simple image-conditioning are required to learn robust and accurate appearance representations.
sections.heading: 4.3 Cross-dataset generalization.
sections.text: We further study the generalization ability of our method across different datasets by training on one dataset and testing on another one as shown in Table. 4a. We would like to point out that the two
  datasets [33, 16] have significantly different statistics both in terms of color distribution (background, lighting) and distance of the camera to the subject, making the cross-dataset generalization task extremely challenging. Nevertheless, we found that only 8-minute fine-tuning on the target dataset can already outperform the baselines fully-trained on the target dataset. 16-minute fine-tuning performs on par with our model fully-trained on the target dataset.
sections.heading: 4.4 Ablation studies
sections.text: Table. 3 shows the ablation study on the ZJU-MoCap dataset for unseen subjects and unseen poses, using three time-steps and three camera views as input. Note that all the variations without either temporal or multi-view transformer modules use simple average pooling instead, to fuse temporal or multi-view observations respectively.
  Complementariness of skeletal and pixel-aligned query features. ‘Sk’ uses only timeaugmented skeletal features (Sec. 3.2) without time-specific pixel-aligned features, while ‘Px’ uses only the time-specific pixel-aligned features, on the contrary. Both ‘only’ models show the largest drops compared to our full model, and ‘Sk + Px’ model improves them by +1.2 PSNR and +0.9 PSNR respectively. This validates the complementariness between the skeletal and pixel-aligned features in that one is time-augmented but involves slight geometric deviations, while another is time-specific and represents exact query location, as discussed in Sec. 3.3.
  Impact of temporal and multi-view transformers. ‘Sk + Px’ uses no transformers so far, falling behind our full model by -1.3 PSNR score. Then ‘Sk + Px + T’ adds the temporal transformer and improves +0.7 PSNR score, showing its effectiveness in aggregating information over video frames. ‘Sk + Px + MV’ uses the multi-view transformer module and shows the largest gain of +1.0 PSNR, indicating the efficacy of learnt cross-attention between the skeletal features and pixel-aligned features, as well as the importance of learnt inter-view correlations. Our full model ‘Sk + Px + T + MV’ shows the best use of all the proposed components and achieves 24.75 PSNR and 0.9058 SSIM.
  Impact of number of camera views. Table. 4b shows the performance of our method when tested with different number of input views, fixing the number of timesteps to one. As expected, the performance degrades slightly with fewer input views (e.g., as few as one).
sections.heading: 5 Limitations
sections.text: While our novel generalizable human performance rendering method outperforms recent per-subject and generalizable NERF methods, there are remaining challenges yet to be explored. 1) While we show that our method can generalize across datasets with finetuning, the generalization capability will be limited as the distribution of the datasets become significantly different. 2) The performance of our method will be affected as the SMPL parameter accuracy degrades. This is the reason for lower performance on the AIST dataset where the accuracy of the SMPL fits are low due to highly complex motion sequences. It is an interesting direction to jointly refine the SMPL parameters using differentiable rendering within our framework. 3) Our algorithm does not have an explicit assumption of static cameras. However, in practice, it might be hard to estimate the inputs to our method (SMPL fits, camera parameters, foreground masks) with moving cameras due to motion blur, changing background, lighting, and synchronization issues etc. We consider this as an orthogonal problem and expect that any advancements in unconstrained multi-view capture setups will help to generalize our method to in-the-wild settings.
sections.heading: 6 Societal impact
sections.text: We discuss the potential societal impact of our work. On a positive note, the human performance synthesis is the key component of realizing telepresence, which has become more important especially in this pandemic era. In the future, people physically apart can feel like they are in the same space and feel connected with a few inexpensive webcams and AR/VR headsets thanks to the development of the telepresence. Like any other technology, our method can also be used for other reasons by corporations, e.g., for identity recognition from a small number of surveillance cameras. We strongly hope that our research will be used in positive directions.
sections.heading: 7 Conclusion
sections.text: We present Neural Human Performer, a generalizable radiance field network based on a parametric body model that can synthesize free-viewpoint videos for arbitrary human performers from sparse camera views. Leveraging the trackable visual features from the input body motion prior, we propose a combination of a temporal and a multi-view Transformer that integrates multi-time and multi-view observations in a feed-forward manner. Our method can produce photo-realistic view synthesis of unseen poses and identities at test time. In various generalization settings on the ZJU-MoCap and AIST datasets, our method achieves state-of-the-art performance compared to other per-subject methods that utilize a human body prior as well as the generalizable NeRF methods.
  Acknowledgments and Disclosure of Funding
  We thank Sida Peng of Zhejiang University, Hangzhou, China, for many very helpful discussions on a variety of implementation details of the Neural Body [33]. We thank Ruilong li and Alex Yu of UC Berkeley for many discussions on the AIST++ dataset [16] and Pixel-NeRF [52] details. We thank Prof. Alex Berg of UNC for the generous offer of computational resources and Misha Shvets of UNC for a useful tutorial on it. This work was partially supported by National Science Foundation Award 1840131.
title: Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering