summaries: 1. This is the first work that attempts to reconstruct 3D shape from 2D image in an unsupervised way using GANs. The idea is neat: Use networks to predict four 3D parameters and use GAN to generate / synthesize the images corresponding to a set of parameters. Then these synthesized images can be used as pseudo ground truth to train the 3D parameter network.

summaries: This paper studies an interesting inverse-graphics problem. It proposed a novel method to learn 3D shape reconstruction using pre-trained 2D image generative adversarial networks. Given an image containing one single object of interest, it first predicts the graphics code (e.g., viewpoint, lighting, depth, and albedo) by minimizing the reconstruction error using a differentiable renderer. The next step is to render many pseudo samples by randomization in the viewpoint and lighting space, while keeping the predicted depth and albedo fixed. A pre-trained 2D image GAN is further used to project the pseudo samples to the learned data manifold through GAN-Inversion. Finally, these projected samples are added to the set for the next round optimization. Experimental evaluations have been conducted on several categories including face, car, building, and horse.

summaries: This paper proposes an iterative method that jointly estimates viewpoints, light directions, depth, and albedo from single images, by projecting intermediate renderings to the nautral image manifold. Intuitively, the method works by generating, with pre-trained GANs, multiple views of the same object under different lightings, and then inferring 3D shapes from those variants. The key idea is to use pre-trained 2D GANs to make such data generation photorealistic. The authors also demonstrate 3D edits, such as 3D rotation and relighting, that one can perform after running their model.

abstractText: Natural images are projections of 3D objects on a 2D image plane. While state-ofthe-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an offthe-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cars, buildings, etc. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code is available at https://github.com/XingangPan/GAN2Shape.

authors:
Rameen Abdal
Yipeng Qin
Peter Wonka

title: Image2stylegan: How to embed images into the stylegan latent space

venue: In ICCV,

authors:
Volker Blanz
Thomas Vetter

title: A morphable model for the synthesis of 3d faces

venue: In Proceedings of the 26th annual conference on Computer graphics and interactive techniques,

authors:
Andrew Brock
Jeff Donahue
Karen Simonyan

title: Large scale gan training for high fidelity natural image synthesis

venue: In ICLR,

authors:
Jiankang Deng
Jia Guo
Niannan Xue
Stefanos Zafeiriou

title: Arcface: Additive angular margin loss for deep face recognition

venue: In CVPR,

authors:
Yu Deng
Jiaolong Yang
Dong Chen
Fang Wen
Xin Tong

title: Disentangled and controllable face image generation via 3d imitative-contrastive learning

authors:
Baris Gecer
Stylianos Ploumpis
Irene Kotsia
Stefanos Zafeiriou

title: Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction

authors:
Shubham Goel
Angjoo Kanazawa
Jitendra Malik

title: Shape and viewpoint without keypoints

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

venue: In NIPS,

authors:
Erik Härkönen
Aaron Hertzmann
Jaakko Lehtinen
Sylvain Paris

title: Ganspace: Discovering interpretable gan controls

venue: arXiv preprint arXiv:2004.02546,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Philipp Henzler
Niloy J Mitra
Tobias Ritschel

title: Escaping plato’s cave: 3d shape from adversarial rendering

authors:
Justin Johnson
Alexandre Alahi
Li Fei-Fei

title: Perceptual losses for real-time style transfer and super-resolution

venue: In ECCV,

authors:
Angjoo Kanazawa
Michael J Black
David W Jacobs
Jitendra Malik

title: End-to-end recovery of human shape and pose

authors:
Angjoo Kanazawa
Shubham Tulsiani
Alexei A Efros
Jitendra Malik

title: Learning categoryspecific mesh reconstruction from image collections

authors:
Tero Karras
Samuli Laine
Timo Aila

title: A style-based generator architecture for generative adversarial networks

authors:
Tero Karras
Miika Aittala
Janne Hellsten
Samuli Laine
Jaakko Lehtinen
Timo Aila

title: Training generative adversarial networks with limited data

venue: In Proc. NeurIPS,

authors:
Tero Karras
Samuli Laine
Miika Aittala
Janne Hellsten
Jaakko Lehtinen
Timo Aila

title: Analyzing and improving the image quality of stylegan

authors:
Hiroharu Kato
Yoshitaka Ushiku
Tatsuya Harada

title: Neural 3d mesh renderer

authors:
Xueting Li
Sifei Liu
Kihwan Kim
Shalini De Mello
Varun Jampani
Ming-Hsuan Yang
Jan Kautz

title: Self-supervised single-view 3d reconstruction via semantic consistency

authors:
Ziwei Liu
Ping Luo
Xiaogang Wang
Xiaoou Tang

title: Deep learning face attributes in the wild

venue: In ICCV, pp

authors:
Matthew Loper
Naureen Mahmood
Javier Romero
Gerard Pons-Moll
Michael J Black

title: Smpl: A skinned multi-person linear model

venue: ACM transactions on graphics (TOG),

authors:
Sebastian Lunz
Yingzhen Li
Andrew Fitzgibbon
Nate Kushman

title: Inverse graphics gan: Learning to generate 3d shapes from unstructured 2d data

venue: arXiv preprint arXiv:2002.12674,

authors:
Andrew L Maas
Awni Y Hannun
Andrew Y Ng

title: Rectifier nonlinearities improve neural network acoustic models

venue: In ICML,

authors:
Thu Nguyen-Phuoc
Chuan Li
Lucas Theis
Christian Richardt
Yong-Liang Yang

title: Hologan: Unsupervised learning of 3d representations from natural images

authors:
Xingang Pan
Xiaohang Zhan
Bo Dai
Dahua Lin
Chen Change Loy
Ping Luo

title: Exploiting deep generative prior for versatile image restoration and manipulation

authors:
Omkar M Parkhi
Andrea Vedaldi
Andrew Zisserman
CV Jawahar

title: Cats and dogs

authors:
Pascal Paysan
Reinhard Knothe
Brian Amberg
Sami Romdhani
Thomas Vetter

title: A 3d face model for pose and illumination invariant face recognition

venue: In 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance,

authors:
Mihir Sahasrabudhe
Zhixin Shu
Edward Bartrum
Riza Alp Guler
Dimitris Samaras
Iasonas Kokkinos

title: Lifting autoencoders: Unsupervised learning of a fully-disentangled 3d morphable model using deep non-rigid structure from motion

venue: In ICCV Workshops,

authors:
Soubhik Sanyal
Timo Bolkart
Haiwen Feng
Michael J Black

title: Learning to regress 3d face shape and expression from an image without 3d supervision

authors:
Jiaxiang Shang
Tianwei Shen
Shiwei Li
Lei Zhou
Mingmin Zhen
Tian Fang
Long Quan

title: Self-supervised monocular 3d face reconstruction by occlusion-aware multi-view geometry consistency

authors:
Yujun Shen
Bolei Zhou

title: Closed-form factorization of latent semantics in gans

venue: arXiv preprint arXiv:2007.06600,

authors:
Yujun Shen
Jinjin Gu
Xiaoou Tang
Bolei Zhou

title: Interpreting the latent space of gans for semantic face editing

authors:
Attila Szabó
Givi Meishvili
Paolo Favaro

title: Unsupervised generative 3d shape learning from natural images

venue: arXiv preprint arXiv:1910.00287,

authors:
Ayush Tewari
Mohamed Elgharib
Gaurav Bharaj
Florian Bernard
Hans-Peter Seidel
Patrick Pérez
Michael Zollhofer
Christian Theobalt

title: Stylerig: Rigging stylegan for 3d control over portrait images

authors:
Luan Tran
Xiaoming Liu

title: Nonlinear 3d face morphable model

authors:
Shubham Tulsiani
Nilesh Kulkarni
Abhinav Gupta

title: Implicit mesh reconstruction from unannotated image collections

venue: arXiv preprint arXiv:2007.08504,

authors:
Jiajun Wu
Chengkai Zhang
Tianfan Xue
Bill Freeman
Josh Tenenbaum

title: Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling

authors:
Shangzhe Wu
Christian Rupprecht
Andrea Vedaldi

title: Unsupervised learning of probably symmetric deformable 3d objects from images in the wild

authors:
Yuanbo Xiangli
Yubin Deng
Bo Dai
Chen Change Loy
Dahua Lin

title: Real or not real, that is the question

authors:
Fisher Yu
Ari Seff
Yinda Zhang
Shuran Song
Thomas Funkhouser
Jianxiong Xiao

title: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop

venue: arXiv preprint arXiv:1506.03365,

authors:
Weiwei Zhang
Jian Sun
Xiaoou Tang

title: Cat head detection-how to effectively exploit shape and texture features

authors:
Hengshuang Zhao
Jianping Shi
Xiaojuan Qi
Xiaogang Wang
Jiaya Jia

title: Pyramid scene parsing network

authors:
Hang Zhou
Jihao Liu
Ziwei Liu
Yu Liu
Xiaogang Wang

title: Rotate-and-render: Unsupervised photorealistic face rotation from single-view images

authors:
Tinghui Zhou
Matthew Brown
Noah Snavely
David G Lowe

title: Unsupervised learning of depth and ego-motion from video

authors:
Jun-Yan Zhu
Philipp Krähenbühl
Eli Shechtman
Alexei A Efros

title: Generative visual manipulation on the natural image manifold

heading: 1 INTRODUCTION

text:
Generative adversarial networks (GANs) (Goodfellow et al., 2014) are capable of modeling the 2D natural image manifold (Zhu et al., 2016) of diverse object categories with high fidelity. Recall the fact that natural images are actually the projections of 3D objects to the 2D plane, an ideal 2D image
manifold should be able to reflect some underlying 3D geometrical properties. For example, it is shown that a GAN could shift the object in its generated images (e.g., human faces) in a 3D rotation manner, as a direction in the GAN image manifold may correspond to viewpoint change (Shen et al., 2020). This phenomenon motivates us to ask - “Is it possible to reconstruct the 3D shape of a single 2D image by exploiting the 3D-alike image manipulation effects produced by GANs?” Despite its potential to serve as a powerful method to learn 3D shape from unconstrained RGB images, this problem remains much less explored.
Some previous attempts (Lunz et al., 2020; Henzler et al., 2019; Szabó et al., 2019) also adopt GANs to learn 3D shapes from images, but they rely on explicitly modeling 3D representation and rendering during training (e.g. 3D voxels, 3D models). Due to either heavy memory consumption or additional training difficulty brought by the rendering process, the qualities of their generated samples notably lag behind their 2D GAN counterparts. Another line of works (Wu et al., 2020; Goel et al., 2020; Tulsiani et al., 2020; Li et al., 2020) for unsupervised 3D shape learning generally learns to infer the viewpoint and shape for each image in an ‘analysis by synthesis’ manner. Despite their impressive results, these methods often assume object shapes are symmetric (symmetry assumption) to prevent trivial solutions, which is hard to generalize to asymmetric objects such as ‘building’.
We believe that existing pre-trained 2D GANs, without above specific designs, already capture sufficient knowledge for us to recover the 3D shapes of objects from 2D images. Since the 3D structure of an instance could be inferred from images of the same instance with multiple viewpoint and lighting variations, our insight is that we may create these variations by exploiting the image manifold captured by 2D GANs. However, the main challenge is to discover well-disentangled semantic directions in the image manifold that control viewpoint and lighting in an unsupervised manner, as manually inspect and annotate the samples in the image manifold is laborious and time-consuming.
To tackle the above challenge, we observe that for many objects such as faces and cars, a convex shape prior like ellipsoid could provide a hint on the change of their viewpoints and lighting conditions. Inspired by this, given an image generated by GAN, we employ an ellipsoid as its initial 3D object shape, and render a number of unnatural images, called ‘pseudo samples’, with various randomly-sampled viewpoints and lighting conditions as shown in Fig. 2. By reconstructing them using the GAN, these pseudo samples could guide the original image towards the sampled viewpoints and lighting conditions in the GAN manifold, producing a number of natural-looking images, called ‘projected samples’. These projected samples could be adopted as the ground truth of the differentiable rendering process to refine the prior 3D shape (i.e. an ellipsoid). To achieve more precise results, we further regard the refined shape as the initial shape and repeat the above steps to progressively refine the 3D shape.
With the proposed approach, namely GAN2Shape, we show that existing 2D GANs trained on images only are sufficient to accurately reconstruct the 3D shape of a single image for many object categories such as human faces, cars, buildings, etc. Our method thus is an effective approach for unsupervised 3D shape reconstruction from unconstrained 2D images without any 2D keypoint or 3D annotations. With an improved GAN inversion strategy, our method works not only for GAN samples, but also for real natural images. On the BFM benchmark (Paysan et al., 2009), our method outperforms a recent strong baseline designed specifically for 3D shape learning (Wu et al., 2020). We also show high-quality 3D-aware image manipulations using the semantic latent directions discovered by our approach, which achieves more accurate human face rotation than other competitors.
Our contributions are summarized as follows. 1) We present the first attempt to reconstruct the 3D object shapes using GANs that are pre-trained on 2D images only. Our work shows that 2D GANs
inherently capture rich 3D knowledge for different object categories, and provides a new perspective for 3D shape generation. 2) Our work also provides an alternative unsupervised 3D shape learning method, and does not rely on the symmetry assumption of object shapes. 3) We achieve highly photo-realistic 3D-aware image manipulations including rotation and relighting without using any external 3D models.

heading: 2 RELATED WORK

text:
Generative Adversarial Networks. Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Xiangli et al., 2020) have achieved great success in 2D natural image modeling. After a series of improvements, the state-of-the-art GANs like StyleGAN (Karras et al., 2019; 2020b) and BigGAN (Brock et al., 2019) could synthesis images with high fidelity and resolution. These GANs are composed of 2D operations, like 2D convolutions. Recently, there is a surge of interest to model 3D-aware image distributions with 3D GAN architectures and 3D representations (Nguyen-Phuoc et al., 2019; Wu et al., 2016; Szabó et al., 2019; Henzler et al., 2019; Lunz et al., 2020). However, due to either heavy memory consumption or increased difficulty in training, there are still gaps between their image qualities and those of 2D GANs.
Therefore, in this work, we are interested in investigating whether we can recover the 3D object shape by directly mining the knowledge of 2D GANs. Our resulting method could be viewed as a memory-efficient way for 3D shape and texture generation, which shares the benefits of both 2D and 3D generative models, e.g., high quality and resolution samples, and a 3D interpretation of contents.
3D-aware Generative Manipulation. Given a pre-trained GAN, there has been works trying to discover the latent space directions that manipulate the image content in a 3D-controllable manner. InterFaceGAN (Shen et al., 2020) learns to rotate human faces by learning latent direction from manual annotations. In contrast, SeFa (Shen & Zhou, 2020) and GANSpace (Härkönen et al., 2020) discover meaningful latent directions without supervision, where some of them is coupled with content pose. Recently, StyleRig (Tewari et al., 2020), DiscoFaceGAN (Deng et al., 2020), and RotateRender (Zhou et al., 2020) achieves 3D-aware manipulation on human face by using an external 3D human face model, i.e., 3DMM (Blanz & Vetter, 1999), as a strong 3D shape prior.
Unlike these methods, our method could accurately manipulate on both pose and lighting factors without borrowing external 3D models or supervisions. This allows our method to generalize beyond human faces, like cats and cars.
Unsupervised 3D Shape Learning. Unsupervised learning of 3D shapes from raw, monocular view images has attracted increasing attention recently, as it relieves the heavy burden on manual annotations. The key challenge of this problem is the lack of multiple view or lighting images for a single instance. To tackle this problem, a number of works leverage external 3D shape models or weak supervisions like 2D key-points. For example, Gecer et al. (2019); Kanazawa et al. (2018a); Sanyal et al. (2019); Shang et al. (2020) reconstruct the 3D shape of human faces or bodies using 3DMM (Blanz & Vetter, 1999) or SMPL (Loper et al., 2015) as a prior knowledge. And Tran & Liu (2018); Kanazawa et al. (2018b) adopt the guidance of 2D key-points to learn the 3D shape of face or objects. The works of Tulsiani et al. (2020); Goel et al. (2020) further discard the need of former supervisions, but still rely on initial category-specific shape templates.
Some representative works that rely only on natural 2D images include Sahasrabudhe et al. (2019); Wu et al. (2020); Li et al. (2020)1, which learn to disentangle the viewpoint and shape for each image via autoencoders that explicitly model the rendering process. However, these methods still rely on certain regularization to prevent trivial solutions. For example, Sahasrabudhe et al. (2019) adopts an autoencoder with a small bottleneck embedding to constrain the deformation field, which tend to produce blur and imprecise results. Wu et al. (2020); Li et al. (2020) adopt the assumption that many object shapes are symmetric, thus their inferred shapes sometimes miss the unsymmetric aspects of the objects in images.
Our method in this work provides an alternative solution for the fore-mentioned ‘single view’ challenge, which is to obtain multiple view and lighting images by exploiting the image manifold modeled by 2D GANs. Our method does not necessarily rely on the symmetry assumption on object
1We include methods that require silhouettes in this part.

heading: Random sample

text: shape, thus better captures the asymmetric aspects of objects. In experiments, we demonstrate superior performance over a recent state-of-the-art method Unsup3d (Wu et al., 2020).

heading: 3 METHODOLOGY

text:
Preliminaries on Generative Adversarial Networks. We first provide some preliminaries on GANs before discussing how we exploit GANs for 3D shape recovery. GANs (Goodfellow et al., 2014) have achieved great successes in modeling natural images. A GAN consists of a generator G that maps a latent vector z to an image x ∈ R3×H×W , and a discriminator D that distinguishes between generated images and real ones. z is drawn from a prior distribution such as the multivariate normal distribution N (0, I)d. GANs trained on natural images are able to continuously sample realistic images, thus could be used as an approximation of the natural image manifold (Zhu et al., 2016).
In this work, our study is mainly based on a representative GAN, StyleGAN2 (Karras et al., 2020b). The generator of StyleGAN2 consists of two parts, a mapping network F : Z 7→ W that maps the latent vector z in the input latent space Z to an intermediate latent vector w ∈ W , and a synthesis network that mapsw to the output image. In the following sections, we refer to the synthesis network as G. The image manifold of StyleGAN2 thus could be defined as M = {G(w)|w ∈ W}. Such a two-stage structure makes it easier to perform GAN-inverson on top of StyleGAN2. i.e., searching for a w that best reconstructs a real target image (Abdal et al., 2019).

heading: 3.1 FROM 2D GANS TO 3D SHAPES

text:
In order to mine the viewpoint and lighting cues in the GAN image manifold, we need to explicitly model these factors. Here we follow the photo-geometric autoencoding design in Wu et al. (2020). For an image I ∈ R3×H×W , we adopt a function that is designed to predict four factors (d,a,v, l),
including a depth map d ∈ RH×W , an albedo image a ∈ R3×H×W , a viewpoint v ∈ R6, and a light direction l ∈ S2. This function thus is implemented with four corresponding sub-networks (D,A, V, L), as shown in Fig. 3 (a). The four factors are recomposed to form the original image I via a rendering process Φ, which consists of lighting Λ and reprojection Π steps as follows:
Î = Φ(d,a,v, l)
= Π(Λ(d,a, l),d,v) (1)
Here Λ performs shading with albedo a, depth map d, and light direction l, while Π performs viewpoint change and generates the image viewed from viewpoint v. The viewpoint change is realized with a differentiable renderer (Kato et al., 2018). We put more details in the Appendix.
Step 1: Using a Weak Shape Prior. As our goal is to explore images of the same instance in the GAN image manifold that have novel viewpoints and lighting conditions, we would like to seek for a hint on the change of viewpoint and lighting to guide the exploration. To achieve this, we make the observation that many objects including faces and cars have a somewhat convex shape prior. We could thus resort to an ellipsoid as a weak prior to create ‘pseudo samples’ with varying viewpoints and lightings as the hint.
Specifically, as shown in Fig.3 (a), given an image sample I from the GAN, we initialize the depth map d to have an ellipsoid shape. With an off-the-shelf scene parsing model (Zhao et al., 2017), we could position the ellipsoid to be roughly aligned with the object in the image. Then the albedo network A is optimized with the reconstruction objective L(I, Î), where Î is calculated via Eq. (1), andL is a weighted combination of L1 loss and perceptual loss (Johnson et al., 2016). The viewpoint v and lighting l are initialized with a canonical setting, i.e., v0 = 0 and lighting is from the front. This allows us to have an initial guess about the four factors d0,a0,v0, l0, with d0 being the weak shape prior.
Step 2: Sampling and Projecting to the GAN Image Manifold. With the above shape prior as an initialization, we are able to create ‘pseudo samples’ by sampling a number of random viewpoints {vi|i = 1, 2, ...,m} and lighting directions {li|i = 1, 2, ...,m}. Specifically, we use {vi} and {li} along with d0 and a0 to render pseudo samples {Ii} via Eq. (1). {vi} and {li} are randomly sampled from some prior distributions, like the multi-variate normal distribution. As shown in Fig.3 (b), although these pseudo samples have unnatural distortions and shadows, they provide cues on how the face rotates and how the light changes.
In order to leverage such cues to guide novel viewpoint and lighting direction exploration in the GAN image manifold, we perform GAN inversion to these pseudo samples, i.e., reconstruct them with the GAN generatorG. Specifically, we train an encoderE that learns to predict the intermediate latent vector wi for each sample. Unlike prior works that directly predict the latent vector, in our setting the latent vector w for the original sample I is known, thus the encoder E only need to predict an offset ∆wi = E(Ii) that encodes the difference between the pseudo sample Ii and the original sample I, which is much easier. Thus the optimization goal is:
θE = arg min θE
1
m m∑ i=0 L ′ ( Ii, G ( E(Ii) +w )) + λ1‖E(Ii)‖2 (2)
where m is the number of samples, θE is the parameter of the encoder E, λ1 is the regularization coefficient, and L′ is a distance metric for images. Following Pan et al. (2020), we adopt the L1 distance of the discriminator features as the distance metric L′ , which is shown to work better for GAN samples. The regularization term ‖E(Ii)‖2 is to prevent the latent offset to grow too large during training.
As shown in Fig.3 (b), to achieve better reconstruction, the projected samples {Ĩi = G(E(Ii)+w)} should have similar viewpoint and lighting changes as the pseudo samples {Ii}. In the meanwhile, the generator G could regularize the projected samples to lie in the natural image manifold, thus fixing the unnatural distortions and shadows in the pseudo samples. In the next step, we discuss how to exploit these projected samples to infer the 3D shape.
Step 3: Learning the 3D Shape. The above projected samples {Ĩi} provide images of multiple viewpoint and lighting conditions while having nearly the same object content, e.g., face identity. Such ‘single instance multiple view&lighting’ setting makes it possible to learn the underlying
3D shape with aforementioned photo-geometric autoencoding model. Specifically, as illustrated in Fig.3 (c), the viewpoint network V and the lighting network L predict instance-specific view ṽi and lighting l̃i for each sample Ĩi, while the depth network D and albedo network A output the shared depth ṽ and albedo ã with the original sample I as input. The four networks are jointly optimized with the following reconstruction objective:
θD,θA,θV ,θL = arg min θD,θA,θV ,θL
1
m m∑ i=0 L ( Ĩi,Φ ( D(I), A(I), V (Ĩi), L(Ĩi) )) + λ2Lsmooth ( D(I) ) (3)
where θD,θA,θV ,θL are the parameters of network D,A, V, L respectively, and Lsmooth ( D(I) ) is a smoothness term defined the same way as in (Zhou et al., 2017). Note that we also reconstruct the original sample I, i.e., use I as an additional Ĩi, to prevent the infered factors to deviate from the original sample. Here we do not use the sampled {vi} and {li} in step 2, as the viewpoints and lighting directions of the projected samples may not accurately match the pseudo samples, thus we learn to predict them with V and L instead.
The above reconstruction loss would decrease only when the underlying depth, albedo, and all samples’ viewpoints and lighting directions are correctly inferred. As shown in Fig.3 (c), the object shape, parameterized by depth, does shift toward the true underlying shape, e.g., shape of a human face. And the unnatural distortions and shadows that exist in the pseudo samples are mitigated in the reconstructions.
Iterative Self-refinement. While the above three steps produce a more accurate 3D object shape than the prior shape, it may miss some details like the wrinkles on the face. This is because the projected samples {Ĩi}may not accurately preserve all semantics of the original instance in I, as the pseudo samples are obtained from an approximated weak shape prior. Fortunately, we can repeat the cycle of these three steps multiple times by regarding the refined 3D shape and trained networks in the previous cycle as the initialized shape and networks in a new cycle. This allows us to refine the 3D shape and projected samples progressively. In our experiments, we use four cycles of these three steps unless otherwise stated.

heading: 3.2 DISCUSSION

text:
Regularizing the Latent Offset. In the Eq. 2 of Step 2, the latent offset ∆wi = E(Ii) is only regularized with a weak L2 norm term, thus the resulting latent vector ∆wi + w may fall out of the intermediate latent distributionW of the GAN. This could possibly cause the projected samples {Ĩi} to inherit some unnatural distortions in the pseudo samples {Ii}, thus stronger regularization on ∆wi is required. To address this issue, we propose a new regularization strategy, which is to restrict ∆wi to be a valid offset created by the mapping network F as ∆wi = F (E(Ii)) − F (0). We provide more discussion on this design in the Appendix.
Joint Training of Multiple Instances. Note that our discussion above involves only a single target image I, i.e., our method is used in an instance-specific manner. However, our pipeline could also be jointly trained on multiple instances to achieve a better generalization ability and a faster convergence speed on new samples. The extension to joint training is straightforward, whose details are included in the Appendix. The key difference between training with multiple instances and a single instance is that, since there are multiple target instances with diverse viewpoint and lighting variations, the absolute latent offset E(Ii) in Eq. 2 should be replaced with a relative latent offset E(Ii)− E(I), which could generalize better across different target instances.

heading: 4 EXPERIMENTS

text: Implementation Details. We first evaluate our method on 3D shape recovery, and then show its application to 3D-aware image manipulation. The datasets used include CelebA (Liu et al., 2015), BFM (Paysan et al., 2009), a combined cat dataset (Zhang et al., 2008; Parkhi et al., 2012), LSUN Car (Yu et al., 2015), and LSUN Church (Yu et al., 2015), all of which are unconstrained RGB images. For BFM, we use the same one as in Wu et al. (2020). We adopt StyleGAN2 pre-trained on these datasets in our experiments. We recommend readers to refer to the Appendix for more implementation details and qualitative results.

heading: 4.1 UNSUPERVISED 3D SHAPE RECONSTRUCTION

text:
Qualitative Evaluation. The qualitative results of our method and Unsup3d (Wu et al., 2020) are shown in Fig.1, Fig.4, and Fig.5. Our method recovers the 3D shapes with high-quality across human faces, cats, cars, and buildings. For example, the wrinkles on human faces and the edges and flat surfaces on cars are well captured. The results of Unsup3d, while good, tend to miss some asymmetric aspects of human faces, e.g., its predicted eye directions always point to the front. Besides, as shown in Fig.4 (b) and Fig.5, Unsup3d has difficulty in generalizing to datasets with asymmetric objects or large viewpoint variations like buildings and cars, while our method does not.
Quantitative Evaluation. Quantitative comparison is conducted on the BFM dataset. Following (Wu et al., 2020), we report the scale-invariant depth error (SIDE) and mean angle deviation (MAD) scores as the evaluation metrics. In order to evaluate on the test set images instead of GAN generated samples, we need to first conduct GAN inversion, i.e., finding the intermediate latent vectors {w} that best reconstruct these images. We adopt the method in Karras et al. (2020b), which achieves satisfactory reconstruction results, as shown in the first row of Fig.6. To achieve better generalization, our method is pre-trained on only 200 training images before testing (the training set has 160k images in total). Due to the additional testing time brought by the GAN inversion and instance-specific training steps, we report results on the first 500 images of the test set.
O u rs
U n su
p 3 d
Although the above GAN inversion step could introduce additional error due to imperfect reconstruction, our method still noticeably outperforms Unsup3d and other trivial baselines, as shown in Tab.1. It is worth noting that our method does not necessarily rely on the symmetric assumption. To demonstrate this, we also report results of our method and Unsup3d when both are learned without the symmetric assumption. As No.(7)(8) in Tab.1 and Fig.6 show, in this setting our method significantly outperforms Unsup3d, which tends to produce messy results. Furthermore, No.(5) provides the results without using the latent offset regularization described in Sec.3.2, where the performance decreases notably, demonstrating the effectiveness of the proposed regularization technique.
We further study the effects of using different shape prior. The setting is the same as Tab.1, and the results are shown in Tab.2. Some variations like asymmetric, shifted, and weaker shape prior only slightly affect the performance, showing that the result is not very sensitive to the shape prior. But the results would get worse for flat shape, as it could not indicate viewpoint and lighting change.

heading: 4.2 3D-AWARE IMAGE MANIPULATION

text:
Object Rotation and Relighting. After training, the recovered 3D shape and the encoder E immediately allow various 3D-aware image manipulation effects. In Fig.7, we show two manipulation effects, including object rotation and relighting. For each effect, we show results rendered using the recovered 3D shape and albedo, and their corresponding projections in the GAN manifold via the encoder E. The results rendered from 3D shapes strictly follow the underlying structure of the object, while their projections in GAN are more photo-realistic. As mentioned before, our method is applicable to real natural images via GAN inversion, and the first row in Fig.7 is an example, where our method also works well.
Table 3: Identity-preserving face rotation. We compare with HoloGAN, GANSpace, and SeFa. The metrics are identity distances measured as angles in the ArcFace feature embeddings.
Method error mean (deg.)↓ error max (deg.)↓ HoloGAN 47.38 69.24 GANSpace 41.17 58.93 SeFa 41.79 60.73 Ours (3D) 28.93 43.02 Ours (GAN) 39.85 57.21
Identity-preserving Face Rotation. We compare our method with other unsupervised methods that also achieve face rotation with GANs, including HoloGAN (Nguyen-Phuoc et al., 2019), GANSpace (Härkönen et al., 2020), and SeFa (Shen & Zhou, 2020). Specifically, for each method, we randomly sample 100 face images, and for each face, we shift its yaw angle from -20 degree to 20 degree, obtaining 20 samples with different poses. A face identity detection model ArcFace (Deng et al., 2019) is then used to evaluate how much the face identities change during rotation. More details are included in the Appendix. Since our method explicitly recover the 3D face shapes, results rendered from these recovered 3D shapes significantly outperform other counterparts, as shown in Tab.3 and Fig.8. In the alternative attempt where we directly operate in the GAN manifold following the projected directions of the recovered 3D shapes, face identities are also better preserved.
Discussion. We have shown that our method is applicable to many object categories. However, for objects with more sophisticated shapes like horses, a simple convex shape prior may not well reflect the viewpoint and lighting variations, and thus the 3D shapes could not be very accurately inferred. The second row of Fig.9 provides one example. Another limitation of our method is that our 3D mesh is parameterized by a depth map, which could not model the back-side shape of objects, as the first row of Fig.9 shows. This could possibly be addressed by adopting a better parameterization of the 3D mesh.Despite these limitations, our method still captures some aspects of the horses’ shapes, like the rough shape of the head and belly, and achieves reasonable relighting effects.

heading: 5 CONCLUSION

text:
We have presented the first method that directly leverages off-the-shelf 2D GANs to recover 3D object shapes from images. We found that existing 2D GANs inherently capture sufficient knowledge to recover 3D shapes for many object categories, including human faces, cats, cars, and buildings. Based on a weak convex shape prior, our method could explore the viewpoint and lighting variations in the GAN image manifold, and exploit these variations to refine the underlying object shape in an iterative manner. We have further shown the application of our method on 3D-aware image manipulations including object rotation and relighting. Our results uncover the potential of 2D GANs in modeling the underlying 3D geometry of 2D image manifold.
Acknowledgment. This research was conducted in collaboration with SenseTime. This work is supported by NTU NAP and A*STAR through the Industry Alignment Fund - Industry Collaboration Projects Grant. This work is also supported by the HK General Research Fund No.27208720.

heading: A APPENDIX

text: In this appendix, we provide more qualitative results and the implementation details.

heading: A.1 QUALITATIVE EXAMPLES

text:
Fig.10 provides more qualitative results of our method, including 3D shape reconstruction and 3Daware image manipulations. The effects of iterative training is shown in Fig.11, where the object shapes become more precise at latter training stages. Fig.12 shows that our method works well without using the symmetry assumption. Fig.13 shows that our method achieves reasonable results for many challenging cases.
Effects of shape prior. Fig.14 shows the qualitative results of using different shape priors. Unlike the results in Tab.2, here joint training is not adopted to better showcase the effects of different shape. When the ellipsoid is shifted by 1/6 image width or is asymmetric, the shape could be progressively refined during the iterative training process. But the results would get worse if the shift is too large or a flat shape is used, because the viewpoint and lighting changes could not be revealed with these shapes.

heading: Input Stage 1 Stage 2 Stage 4 Stage 1 Stage 2 Stage 4

text:
Effects of dataset size. Fig.4 shows the effects of dataset size used for training the GAN. It can be seen that the performance only drops marginally even when the dataset size is reduced from 160k to 3k. And the performance get notably worse when the dataset size reduces to 1k, as the image quality of the GAN starts to deteriorate given insufficient data. This could potentially be addressed by recent data augmentation strategy for GANs (Karras et al., 2020a).
A.2 IMPLEMENTATION DETIALS

heading: A.2.1 PHOTO-GEOMETRIC AUTOENCODING

text:
The photo-geometric autoencoding model we used mainly follows the work of Wu et al. (2020). Here we describe the Λ and Π in Eq.1.
J = Λ(d,a, l) produces the illuminated texture with the albedo a, depth map d, and light direction l. Specifically, we first calculate the normal map n of the depth map d, which describes the normal direction of the surface for each pixel position (i, j). Then the illuminated texture is calculate with Jij = (ks + kd max 0, 〈l, nij〉) · aij , where ks and kd are the scalar weight for the ambient and diffuse terms, and are predicted by the lighting network V . The lighting direction is described as l = (lx, ly, 1) T /(l2x + l 2 y + 1) 0.5, where lx and ly are also the predictions of the lighting network.
Given the above illuminated texture J, the reprojection function Π would map it to a new viewpoint v as Î = Π(J,d,v). To do this, we denote the 3D point in the reference frame of the camera as P = (Px, Py, Pz) ∈ R3, which is projected to pixel p = (i, j, 1) via:
p ∝KP , K = [ f 0 ci 0 f cj 0 0 1 ] ,  ci = W−1 2 , cj = H−1 2 ,
f = W−1 2 tan
θFOV 2
(4)
where θFOV is the field of view of the perspective camera, and is assume to be θFOV ≈ 10◦. Given a depth map dwhich describes a depth value dij for each pixel (i, j), we would haveP = dij ·K−1p.
The viewpoint v ∈ R6 represents the rotation angles (v1:3) and translations (v4:6) along x, y and z axes. v thus describes an Euclidean transform (R,T ) ∈ SE(3), which transforms 3D points to the predicted viewpoint. The mapping from a pixel (i, j) in the original view to the pixel (i′, j′) in the
predicted view could then be described by the warping function ηd,v : (i, j) 7→ (i′, j′) as:
p′ ∝K(dij ·RK−1p+ T ) (5)
where p′ = (i′, j′, 1). Finally, Π applies this warp to the illuminated texture J to obtain the reprojected image Î = Π(J,d,v) as Îi′j′ = Jij , where (i, j) = η−1d,v(i
′, j′). Here η−1d,v is the backward warping of ηd,v , which is achieved via a differentiable renderer Kato et al. (2018) in the same way as Wu et al. (2020).

heading: A.2.2 REGULARIZING THE LATENT OFFSET

text:
As described in Section 3.2, in order to keep the projected samples {Ĩi} in the natural image manifold, stronger regularization on the predicted latent offset ∆wi is needed. One possible solution is to let the encoder predict the offset in the original latent space, i.e., the one before the mapping network F , then the output of the generator would be G ( F (E(Ii) + z) ) , where z is the original latent vector corresponding to I. However, one issue of this solution is that, compared with the intermediate latent spaceW , it is much more difficult to perform GAN inversion to the original latent space Z to obtain the latent vector z of a real natural image. This would impede this method to be applied to real natural images other than GAN samples.
To address this issue, in this work we propose a new regularization strategy for the latent offset ∆wi, which is to restrict it to be a valid offset created by the mapping network F as follows:
∆wi = F (∆zi)− F (0) = F (E(Ii))− F (0) (6)
which regards ∆wi as the shift in theW space by adding perturbation around zero in the Z space. Note that the point zero is a natural choice as it is the mean point in the prior distribution N (0, I)d. In order to make the regularization strength adjustable, one can further view the mapping network F as two consecutive parts F = F1 ◦ F2, and regularize ∆wi with only F1 as below:
∆wi = F1(E(Ii) + F2(0))− F (0) (7)
The depth d of F1 thus controls the strength of regularization. As shown in Tables 10 11 12, for the BFM dataset, we use the full mapping network (i.e., d = 8). For other datasets with more complex images, the regularization of a full mapping network is too strong and sometimes lead to sub-optimal results, we thus set d = 2.

heading: A.2.3 TRAINING DETIALS

text:
In this section, we provide the network architectures, hyper-parameters, and other details in our experiments.
Model Architectures. For a fair comparison on the BFM dataset, the images used in the photogeometric model are resized to 642 resolution, and the D,A,L, V network have the same architecture as Wu et al. (2020). For other datasets, the image resolution is resized to 1282, and the network architectures are added by additional layers accordingly, as shown in Tab.5 and Tab.6 . For the GAN encoder net E, we adopt a ResNet-alike architecture (He et al., 2016), as presented in Tab.7 and Tab.8 . The abbreviations for the network layers are described below:
Conv(cin, cout, k, s, p): convolution with cin input channels, cout output channels, kernel size k, stride s, and padding p.
Deconv(cin, cout, k, s, p): deconvolution with cin input channels, cout output channels, kernel size k, stride s, and padding p.
GN(n): group normalization (Wu & He, 2018) with n groups.
LReLU(α): leaky ReLU (Maas et al., 2013) with a negative slope of α.
Upsample(s): nearest-neighbor upsampling with a scale of s.
Avg pool(s): average pooling with a stride of s.
ResBlock(cin, cout): residual block as defined in Tab.8.
Table 5: Network architecture for viewpoint net V and lighting net L. The output channel size cout is 6 for V and 4 for L.
Encoder Output size Conv(3, 32, 4, 2, 1) + ReLU 64 Conv(32, 64, 4, 2, 1) + ReLU 32 Conv(64, 128, 4, 2, 1) + ReLU 16 Conv(128, 256, 4, 2, 1) + ReLU 8 Conv(256, 512, 4, 2, 1) + ReLU 4 Conv(512, 512, 4, 1, 0) + ReLU 1 Conv(512, cout, 1, 1, 0) + Tanh 1
Table 6: Network architecture for depth net D and albedo net A. cout is 1 for D and 3 for A.
Encoder Output size Conv(3, 32, 4, 2, 1) + GN(8) + LReLU(0.2) 64 Conv(32, 64, 4, 2, 1) + GN(16) + LReLU(0.2) 32 Conv(64, 128, 4, 2, 1) + GN(32) + LReLU(0.2) 16 Conv(128, 256, 4, 2, 1) + GN(64) + LReLU(0.2) 8 Conv(256, 512, 4, 2, 1) + LReLU(0.2) 4 Conv(512, 256, 4, 1, 0) + ReLU 1 Decoder Output size Deconv(256,512,4,1,0) + ReLU 4 Conv(512,512,3,1,1) + ReLU 4 Deconv(512,256,4,2,1) + GN(64) + ReLU 8 Conv(256,256,3,1,1) + GN(64) + ReLU 8 Deconv(256,128,4,2,1) + GN(32) + ReLU 16 Conv(128,128,3,1,1) + GN(32) + ReLU 16 Deconv(128,64,4,2,1) + GN(16) + ReLU 32 Conv(64,64,3,1,1) + GN(16) + ReLU 32 Deconv(64,32,4,2,1) + GN(8) + ReLU 64 Conv(32,32,3,1,1) + GN(8) + ReLU 64 Upsample(2) 128 Conv(32,32,3,1,1) + GN(8) + ReLU 128 Conv(32,32,5,1,2) + GN(8) + ReLU 128 Conv(32,cout,5,1,2) + Tanh 128
Table 7: Network architecture of GAN encoder net E for 1282 resolution input images. For 642 resolution input, the last ResBlock is removed and the following channels are cut down by a half.
Encoder Output size Conv(3, 32, 4, 2, 1) + ReLU 64 ResBlock(32, 64) 32 ResBlock(64, 128) 16 ResBlock(128, 256) 8 ResBlock(256, 512) 4 Conv(512, 1024, 4, 1, 0) + ReLU 1 Conv(1024, 512, 1, 1, 0) 1
Table 10: Hyper-parameters for LSUN Car and LSUN Church datasets.
Hyper-parameters. The hyper-parameters of our experiments are provided in Tables 9 10 11 12. The batch size here is the batch size used in the Step 2 and Step 3 of our method. And the number of stages indicates how many times the Step 1-3 are repeated.
Joint pre-training. Note that for the CelebA, Cat, and BFM dataset, we adopt joint pre-training on 200 samples to achieve better generalization when fine-tuning on new samples. Specifically, for each iteration of each step, we randomly choose a batch of samples from the 200 samples, and the training is the same as that of a single sample except that the gradient calculated from each sample are averaged after back-propagation.
Symmetry assumption. In our method, the symmetry assumption on object shapes is an optional choice, and could further improve the results for human face and cat datasets. Thus we also adopt this assumption for CelebA, Cat, and BFM dataset unless otherwise stated. Note that the results
without the symmetry assumption are also provided in Figures 6 12 , Tab.12 and other datasets, which is also satisfactory. Unlike Unsup3d (Wu et al., 2020), our method does not need to predict an additional confidence map for the symmetry, as we can drop the symmetry assumption after the first training stage, as shown in Tab.11 .
Sampling viewpoints and lighting directions. In the Step 2 of our method, we need to randomly sample a number of viewpoints and lighting directions from some prior distributions. For viewpoint, we calculate the mean and covariance statistics of viewpoints estimated from the BFM, CelebA, and Cat dataset using the viewpoint prediction network of (Wu et al., 2020). These statistics are used to sample the random viewpoints from multi-variate normal distributions for the corresponding datasets. For LSUN Car and Church datasets, we simply use the statistics estimated from the FFHQ dataset (Karras et al., 2019), and we found it works well.
For lighting directions, we sample from the following uniform distributions: lx ∼ U(xmin, xmax), ly ∼ U(ymin, ymax), kd ∼ U(dmin, dmax), and ks = αkd, where [xmin, xmax, ymin, ymax, dmin, dmax, α] = [−0.9, 0.9,−0.3, 0.8,−0.1, 0.7,−0.4] for BFM dataset and [−1, 1,−0.2, 0.8,−0.1, 0.6,−0.6] for other datasets. Note the sampled kd and ks are added to the predictions of the lighting network L as the final diffuse and ambient term. This setting tends to increase the diffuse term and decrease the ambient term, as the diffuse term could better reflect the structure of the objects.
Masking out the background. In this work, we care about the main object rather than the background. Thus, for GANs trained on the LSUN Car and LSUN Church datasets that contain large background area, we could use an off-the-shelf scene parsing model PSPNet50 (Zhao et al., 2017) to parse the object areas as a pre-processing for better visualization. For an input image, the scene parsing model is used only once to get the object mask for the original view, thus it does not provide signals on the 3D shape. For novel views, the mask is warped with the current shape accordingly, and is used to mask out the image background in Eq.2 and Eq.3 . For other datasets we do not mask out the background.
Face rotation evaluation. In the identity-preserving face rotation experiments, we use a face identity detection model ArcFace (Deng et al., 2019) to evaluate the shift of face identity during rotation. For each human face sample, we first rotate its yaw angle from -20 degree to 20 degrees to obtain 20 samples with different poses. The face angle is obtained with the viewpoint prediction network from Wu et al. (2020). We then calculate the identity distances between the front-view face and other faces, and report both their mean and max values in Tab.11 . For face identity prediction model, the identity distances are represented as angles between the final feature embeddings. The above mean and max values are averaged across 100 samples as the final results.

title: DO 2D GANS KNOW 3D SHAPE? UNSUPERVISED 3D SHAPE RECONSTRUCTION FROM 2D IMAGE GANS


summaries: The majority of feature extraction backbone is shared among different agents and the classifiers of experts are trained with both classification loss and proposed distribution-aware diversity loss. For the second stage, an expert assignment module is trained to re-weight the expert decisions. The whole paper is generally well-organized. However, there are some technical issues authors should further address:

summaries: This paper proposes a Routing Diverse Experts (RIDE) framework to solve the long-tailed classification problem. It has 1) a shared low-level feature extractor and multiple expert classifiers, 2) a distribution-aware diversity loss to encourage experts learning different classification strategies, 3) an expert routing module that dynamically selects a subset of experts for each test instance to make a joint decision. This paper firstly increases the performances on all three splits (many-/med-/few-shot), while most of the existing methods have to sacrifice the head for tail improvements.

summaries: This paper proposes a method termed RoutIng Diverse Experts (RIDE) for reducing both the bias and the variance of a long-tailed classifier. Specifically, RIDE consists of three crucial components: 1) a shared architecture for multiple experts; 2) a distribution-aware diversity loss that encourages more diverse decisions for classes with fewer training instances; 3) an expert routing module that dynamically assigns more ambiguous instances to additional experts. Experiments are conducted on three long-tailed benchmark datasets, i.e., CIFAR100-LT, ImageNet-LT, and iNaturalist. Satisfactory classification results of long-tailed visual recognition are observed.

abstractText: Natural data are often long-tail distributed over semantic classes. Existing recognition methods tackle this imbalanced classification by placing more emphasis on the tail data, through class re-balancing/re-weighting or ensembling over different data groups, resulting in increased tail accuracies but reduced head accuracies. We take a dynamic view of the training data and provide a principled model bias and variance analysis as the training data fluctuates: Existing long-tail classifiers invariably increase the model variance and the head-tail model bias gap remains large, due to more and larger confusion with hard negatives for the tail. We propose a new long-tailed classifier called RoutIng Diverse Experts (RIDE). It reduces the model variance with multiple experts, reduces the model bias with a distribution-aware diversity loss, reduces the computational cost with a dynamic expert routing module. RIDE outperforms the state-of-the-art by 5% to 7% on CIFAR100-LT, ImageNet-LT and iNaturalist 2018 benchmarks. It is also a universal framework that is applicable to various backbone networks, long-tailed algorithms, and training mechanisms for consistent performance gains. Our code is available at: https://github.com/frank-xwang/RIDE-LongTailRecognition.

authors:
Luca Bertinetto
João F Henriques
Jack Valmadre
Philip Torr
Andrea Vedaldi

title: Learning feedforward one-shot learners

venue: In NeurIPS,

authors:
Kaidi Cao
Colin Wei
Adrien Gaidon
Nikos Arechiga
Tengyu Ma

title: Learning imbalanced datasets with label-distribution-aware margin loss

venue: In Advances in Neural Information Processing Systems,

authors:
Zitian Chen
Yanwei Fu
Yu-Xiong Wang
Lin Ma
Wei Liu
Martial Hebert

title: Image deformation meta-networks for one-shot learning

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Peng Chu
Xiao Bian
Shaopeng Liu
Haibin Ling

title: Feature space augmentation for long-tailed data

venue: arXiv preprint arXiv:2008.03673,

authors:
Yin Cui
Menglin Jia
Tsung-Yi Lin
Yang Song
Serge Belongie

title: Class-balanced loss based on effective number of samples

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

title: A unified bias-variance decomposition

venue: In Proceedings of 17th International Conference on Machine Learning,

authors:
Chelsea Finn
Pieter Abbeel
Sergey Levine

title: Model-agnostic meta-learning for fast adaptation of deep networks

venue: arXiv preprint arXiv:1703.03400,

authors:
Raia Hadsell
Sumit Chopra
Yann LeCun

title: Dimensionality reduction by learning an invariant mapping

authors:
Bharath Hariharan
Ross Girshick

title: Low-shot visual recognition by shrinking and hallucinating features

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Haibo He
Edwardo A Garcia

title: Learning from imbalanced data

venue: IEEE Transactions on knowledge and data engineering,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Geoffrey Hinton
Oriol Vinyals
Jeff Dean

title: Distilling the knowledge in a neural network

venue: arXiv preprint arXiv:1503.02531,

authors:
Muhammad Abdullah Jamal
Matthew Brown
Ming-Hsuan Yang
Liqiang Wang
Boqing Gong

title: Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective

authors:
Bingyi Kang
Saining Xie
Marcus Rohrbach
Zhicheng Yan
Albert Gordo
Jiashi Feng
Yannis Kalantidis

venue: In International Conference on Learning Representations,

authors:
Jaehyung Kim
Jongheon Jeong
Jinwoo Shin

title: M2m: Imbalanced classification via major-tominor translation

title: Learning multiple layers of features from tiny images

venue: Technical report,

authors:
Tsung-Yi Lin
Priya Goyal
Ross Girshick
Kaiming He
Piotr Dollár

title: Focal loss for dense object detection

venue: In Proceedings of the IEEE international conference on computer vision,

authors:
Bo Liu
Xudong Wang
Mandar Dixit
Roland Kwitt
Nuno Vasconcelos

title: Feature space transfer for data augmentation

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Jialun Liu
Yifan Sun
Chuchu Han
Zhaopeng Dou
Wenhui Li

title: Deep representation learning on long-tailed data: A learnable embedding augmentation perspective

authors:
Ziwei Liu
Zhongqi Miao
Xiaohang Zhan
Jiayun Wang
Boqing Gong
Stella X Yu

title: Largescale long-tailed recognition in an open world

authors:
Hyun Oh Song
Yu Xiang
Stefanie Jegelka
Silvio Savarese

title: Deep metric learning via lifted structured feature embedding

authors:
Nikolaos Passalis
Anastasios Tefas

title: Learning deep representations with probabilistic knowledge transfer

venue: In Proceedings of the European Conference on Computer Vision,

authors:
Sachin Ravi
Hugo Larochelle

title: Optimization as a model for few-shot learning

authors:
Adam Santoro
Sergey Bartunov
Matthew Botvinick
Daan Wierstra
Timothy Lillicrap

title: Metalearning with memory-augmented neural networks

authors:
Eli Schwartz
Leonid Karlinsky
Joseph Shtok
Sivan Harary
Mattias Marder
Rogerio Feris
Abhishek Kumar
Raja Giryes
Alex M Bronstein

title: Delta-encoder: an effective sample synthesis method for few-shot object recognition

venue: arXiv preprint arXiv:1806.04734,

authors:
Jake Snell
Kevin Swersky
Richard Zemel

title: Prototypical networks for few-shot learning

authors:
Jingru Tan
Changbao Wang
Buyu Li
Quanquan Li
Wanli Ouyang
Changqing Yin
Junjie Yan

title: Equalization loss for long-tailed object recognition

authors:
Yonglong Tian
Dilip Krishnan
Phillip Isola

title: Contrastive representation distillation

authors:
Frederick Tung
Greg Mori

title: Similarity-preserving knowledge distillation

authors:
Grant Van Horn
Oisin Mac Aodha
Yang Song
Yin Cui
Chen Sun
Alex Shepard
Hartwig Adam
Pietro Perona
Serge Belongie

title: The iNaturalist species classification and detection dataset

authors:
Oriol Vinyals
Charles Blundell
Tim Lillicrap
Daan Wierstra

title: Matching networks for one shot learning

authors:
Yu-Xiong Wang
Deva Ramanan
Martial Hebert

title: Learning to model the tail

authors:
Yu-Xiong Wang
Ross Girshick
Martial Hebert
Bharath Hariharan

title: Low-shot learning from imaginary data

authors:
Tong Wu
Qingqiu Huang
Ziwei Liu
Yu Wang
Dahua Lin

title: Distribution-balanced loss for multi-label classification in long-tailed datasets

authors:
Zhirong Wu
Yuanjun Xiong
Stella X Yu
Dahua Lin

title: Unsupervised feature learning via nonparametric instance discrimination

authors:
Liuyu Xiang
Guiguang Ding
Jungong Han

title: Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification

authors:
Saining Xie
Ross Girshick
Piotr Dollár
Zhuowen Tu
Kaiming He

title: Aggregated residual transformations for deep neural networks

authors:
Flood Sung Yongxin Yang
Li Zhang
Tao Xiang
Philip HS Torr
Timothy M Hospedales

title: Learning to compare: Relation network for few-shot learning

authors:
Hongguang Zhang
Jing Zhang
Piotr Koniusz

title: Few-shot learning via saliency-guided hallucination of samples

authors:
Xiao Zhang
Zhiyuan Fang
Yandong Wen
Zhifeng Li
Yu Qiao

title: Range loss for deep face recognition with long-tailed training data

authors:
Boyan Zhou
Quan Cui
Xiu-Shen Wei
Zhao-Min Chen

title: BBN: Bilateral-branch network with cumulative learning for long-tailed visual recognition

authors:
Linchao Zhu
Yi Yang

title: Inflated episodic memory with region self-attention for long-tailed visual recognition

heading: 1 INTRODUCTION

text:
Real-world data are often long-tail distributed over semantic classes: A few classes contain many instances, whereas most classes contain only a few instances. Long-tailed recognition is challenging, as it needs to handle not only a multitude of small-data learning problems on the tail classes, but also extreme imbalanced classification over all the classes.
There are two ways to prevent the many head instances from overwhelming the few tail instances in the classifier training objective: 1) class re-balancing/re-weighting which gives more importance to tail instances (Cao et al., 2019; Kang et al., 2020; Liu et al., 2019), 2) ensembling over different data distributions which re-organizes long-tailed data into groups, trains a model per group, and then combines individual models in a multi-expert framework (Zhou et al., 2020; Xiang et al., 2020).
We compare three state-of-the-art (SOTA) long-tail classifiers against the standard cross-entropy (CE) classifier: cRT and τ -norm (Kang et al., 2020) which adopt a two-stage optimization, first representation learning and then classification learning, and LDAM (Cao et al., 2019), which is trained end-to-end with a marginal loss. In terms of the classification accuracy, a common metric for model selection on a fixed training set, Fig. 1a shows that, all these existing long-tail methods increase the overall, medium- and few-shot accuracies over CE, but decrease the many-shot accuracy.
These intuitive solutions and their experimental results seem to suggest that there is a head-tail performance trade-off in long-tailed recognition. We need a principled performance analysis approach that could shed light on such a limitation if it exists and provide guidance on how to overcome it.
Our insight comes from a dynamic view of the training set: It is merely a sample set of some underlying data distribution. Instead of evaluating how a long-tailed classifier performs on the fixed training set, we evaluate how it performs as the training set fluctuates according to the data distribution.
Published as a conference paper at ICLR 2021Under review as a conference paper at ICLR 2021
Natural data are often long-tail distributed over semantic classes. Existing recog-
nition methods tend to focus on tail performance gain, often at the expense of
head performance loss from increased classifier variance. The low tail perfor-
mance manifests itself in large between-class confusion and high classifier vari-
ance. We aim to reduce both the bias and the variance of a long-tailed classifier
by RoutIng Diverse Experts (RIDE). It has three components: 1) a shared archi-
tecture for multiple classifiers (experts); 2) a distribution-aware diversity loss that
encourages more diverse decisions for classes with fewer training instances; and 3) an expert routing module that dynamically assigns more ambiguous instances to additional experts. With on-par computational complexity, RIDE significantly outperforms the state-of-the-art methods by 5% to 7% on all the benchmarks including CIFAR100-LT, ImageNet-LT and iNaturalist. RIDE is also a universal framework that can be applied to different backbone networks and integrated into various re-balancing or re-weighting methods for consistent performance gains.
1 INTRODUCTION
The natural data we encounter in practice often has a long tail distribution: A few classes contain many instances, while most classes contain only a few instances. Learning discrimination among them is challenging, as the few tail instances can be easily overwhelmed by many head instances.
Long-tailed recognition is usually handled either by class re-balancing/re-weighting strategies giving more importance to tail instances (Cao et al., 2019; Kang et al., 2020; Liu et al., 2019), or by multiexpert methods, where long-tailed data are separated into parts by their frequencies and models focusing on individual parts are combined (Zhou et al., 2020; Xiang & Ding, 2020). However, all these methods generally gain on tail classes at the cost of performance loss on head classes.
The state-of-the-art (SOTA) methods on iNaturalist (Van Horn et al., 2018) are cRT and ⌧ -norm (Kang et al., 2020) and BBN (Zhou et al., 2020). The former belongs to the re-balancing type with a two-stage optimization for learning a good representation and classifier, whereas the latter belongs to the multi-expert type with two experts focusing on head and tail classes.
We analyze the performance of a long-tail classifier in terms of bias and variance with respect to fluctuations in the training set: We randomly sample CIFAR100 (Krizhevsky, 2009) according to a long-tailed distribution a few times, train a model each time, and then estimate the per-class bias and variance of the classifier.
1
Consider the training data D as a random variable. The prediction error of model h on instance x with output Y varies with the realization of D. The expected variance with respect to variable D has a well-known bias-variance decomposition:
Error(x;h) = E[(h(x;D)− Y )2] = Bias(x;h) + Var ance(x;h) + irreducible error(x). (1) For the above L2 loss on regression h(x)→ Y , the model bias measures the accuracy of the prediction with respect to the true value, the variance measures the stability of the prediction, and the irreducible error measures the precision of the prediction and is irrelevant to the model h.
Empirically, for n random sample sets of data, D(1), . . . , D(n), the k-th model trained on D(k) predicts y(k) on instance x, and collectively they have a mean prediction ym. For the L2 regression loss, the model bias is simply the L2 loss between ym and ground-truth t=E[Y ], whereas the model variance is the variance of y(k) with respect to their mean ym:
L2 regression loss: L(y; z) = (y − z)2 (2)
mean prediction: ym = 1
n n∑ k=1 y(k) = arg min z ED[L (h(x); z)] (3)
model bias: Bias(x;h) = (ym − t)2 =L (ym; t) (4)
model variance: Variance(x;h) = 1
n n∑ k=1 ( y(k) − ym )2 =ED[L (h(x); ym)]. (5)
As shown on the above right, these concepts can be expressed entirely in terms of L2 loss L. We can thus extended them to classification (Domingos, 2000) by replacing L with L0-1 for classification:
0-1 classification loss: L0-1(y; z) = 0 if y = z, and 1 otherwise. (6) The mean prediction ym minimizes ∑n k=1 L0-1 ( y(k); ym ) and becomes the most often or main pre-
diction. The bias and variance terms become L0-1(ym; t) and 1n ∑n k=1 L0-1(y(k); ym) respectively.
We apply such bias and variance analysis to the CE and long-tail classifiers. We sample CIFAR100 (Krizhevsky, 2009) according to a long-tail distribution multiple times. For each method, we train
a model per long-tail sampled dataset and then estimate the per-class bias and variance over these multiple models on the balanced test set of CIFAR100-LT Liu et al. (2019). Fig. 1a shows that:
1. On the model bias: The head bias is significantly smaller than the tail bias, at 0.3 vs. 0.9 for CE. All the existing long-tail methods reduce the overall bias by primarily reducing the tail bias. However, the head-tail bias gap remains large at 0.3 vs. 0.8.
2. On the model variance: All the existing long-tail methods increase the model variance across all class splits, with a slight reduction in the medium-shot variance for cRT.
That is, existing long-tail methods reduce the model bias for the tail at the cost of increased model variance for all the classes, and the head-tail model bias gap remains large.
We conduct further statistical analysis to understand the head-tail model bias gap. We examine the largest softmax score in the other classes of {c : c 6= t}, where t is the ground-truth class of an instance. The smaller this hardest negative score is, the less the confusion, and the lower the model bias. Fig. 1b shows that there is increasingly more and larger confusion from the head to the tail.
Guided by our model bias/variance and confusion pattern analysis, we propose a new long-tail classifier with four distinctive features: 1) It reduces the model variance for all the classes with multiple experts. 2) It reduces the model bias for the tail with an additional distribution-aware diversity loss. 3) It reduces the computational complexity that comes with multiple experts with a dynamic expert routing module which deploys another trained distinctive expert for a second (or third, ...) opinion only when it is called for. 4) The routing module and a shared architecture for experts of reduced complexity effectively cut down the computational cost of our multi-expert model, to a level that could be even lower than the commonly adopted baseline with the same backbone.
Our so-called RoutIng Diverse Experts (RIDE) not only reduces the model variance for all the classes, but also significantly reduces the model bias for the tail classes and increases the mean accuracies for all class splits, all of which existing long-tail methods fail to accomplish.
RIDE delivers 5%∼7% higher accuracies than the current SOTA methods on CIFAR100-LT, ImageNet-LT (Liu et al., 2019) and iNaturalist (Van Horn et al., 2018). RIDE is also a universal framework that can be applied to different backbone networks for improving existing long-tail algorithms such as focal loss (Lin et al., 2017), LDAM (Cao et al., 2019), τ -norm (Kang et al., 2020).

heading: 2 RELATED WORKS

text:
Few-shot learning. To generalize from small training data, meta-learning (Bertinetto et al., 2016; Ravi & Larochelle, 2017; Santoro et al., 2016; Finn et al., 2017; Yang et al., 2018) and data augmentation/generation are two most studied approaches (Chen et al., 2019; Schwartz et al., 2018; Zhang et al., 2019; Liu et al., 2018). Matching Network (Vinyals et al., 2016) and Prototypical Network (Snell et al., 2017) learn discriminative features that can be transferred to new classes through meta-learners without big training data. Hariharan & Girshick (2017), Wang et al. (2018) and Liu et al. (2018) utilize samples from a generative model to augment the training data. However, fewshot learning relies on balanced training data, whereas long-tail recognition has to deal with highly imbalanced training data, e.g., from hundreds in the head to a few instances in the tail.
Re-balancing/re-weighting. A direct approach to achieve sample balance is to under- or oversample training instances according to their class sizes (He & Garcia, 2009). Another option is data augmentation, where additional samples are generated to supplement tail classes, sometimes directly in the feature space (Liu et al., 2020; Chu et al., 2020; Kim et al., 2020). Re-weighting modifies the loss function and puts larger weights on tail classes (Lin et al., 2017; Cui et al., 2019; Cao et al., 2019; Wu et al., 2020) or randomly ignoring gradients from head classes (Tan et al., 2020). However, both sample-wise and loss-wise balancing focus on tail classes, resulting in more sensitivity to fluctuations in the small tail classes and thus much increased model variances (Fig. 1a).
Knowledge transfer. OLTR (Liu et al., 2019) and inflated memory (Zhu & Yang, 2020) use memory banks to store and transfer mid- and high-level features from head to tail classes, enhancing feature generalization for the tail. However, this line of work (Liu et al., 2019; Zhu & Yang, 2020; Kang et al., 2020; Jamal et al., 2020; Wang et al., 2017) usually does not have effective control over the knowledge transfer process, often resulting in head performance loss.
Ensembling and grouping. One way to counter imbalance is to separate training instances into different groups based their class sizes. Models trained on individual groups are ensembled together in a multi-expert framework. BBN (Zhou et al., 2020) adaptively fuses two-branches that each focus on the head and the tail respectively. LFME (Xiang et al., 2020) distills multiple teacher models into a unified model, each teacher focusing on a relatively balanced group such as many-shot classes, medium-shot classes, and few-shot classes. BBN and LFME still lose head performance and overall generalizability, as no expert has a balanced access to the entire dataset.
Our RIDE is a non-traditional ensemble method. 1) Its experts have shared earlier layers and reduced later channels, less prone to small tail overfitting. 2) Its experts are jointly optimized. 3) It deploys experts on an as-needed basis for individual instances with a dynamic expert assignment module. 4) It reaches higher accuracies with a smaller model complexity and computational cost.

heading: 3 RIDE: ROUTING DIVERSE DISTRIBUTION-AWARE EXPERTS

text:
We propose a novel multi-expert model (Fig. 2) with shared earlier layers fθ and n independent channel-reduced later layers Ψ = {ψθ1 , ..., ψθn}. They are jointly optimized at Stage 1 and dynamically deployed with a learned expert assignment module at Stage 2. At the inference time, all the m active experts are averaged together in their logits for final ensemble softmax classification:
p = softmax
( 1
m m∑ i=1 ψθi (fθ(x))
) . (7)
Softmax of the average logits is equivalent to the product of individual classification probabilities, which approximates their joint probability if individual experts makes independent decisions.
Experts with a shared early backbone and reduced later channels. Consider n independent experts of the same convolutional neural network (CNN) architecture. Since early layers of a CNN tend to encode generic low-level features, we adopt the common practice in transfer learning and have all the n experts share the same backbone fθ. Each expert retains independent later layers ψθi , i = 1,. . . ,n. To reduce overfitting to small training data in the tail classes, we reduce the number of filter channels in ψθi , e.g., by 1/4. All these n experts are trained together on long-tailed data distribution-aware diversity loss LD-Diversify and classification loss LClassify, such as CE and LDAM. Individual expert classification loss. One way to combine multiple experts is to apply the classification loss to the aggregated logits of individual experts. While this idea works for several recently proposed multi-expert models (Zhou et al., 2020; Xiang et al., 2020), it does not work for our shared experts: Its performance is on-par with an equal-sized single-expert model. Let L denote the classi-
𝑓!
𝜓!!
𝜓!"
𝜓!#
a) Stage 1: Jointly Optimize Diverse Distribution-aware Experts
𝑓!
𝜓!!
𝜓!"
𝜓!#
b) Stage 2: Routing Diverse Experts
Distribution-aware diversity loss ℒ!"#$%&'($)* + Classification loss
ℒ+,-(($).
trainable
frozen
on/off
on/off
Routing loss ℒ/012$34
c) RIDE vs. SOTA
10.8
5.5 6.1
13.8
4.6
10.9
Figure 2: RIDE learns experts and their router in two stages. a) We first jointly optimize multiple experts with individual classification losses and mutual distribution-aware diversity losses. b) We then train a router that dynamically assigns ambiguous samples to additional experts on an as-needed basis. The distribution of instances seen by each expert shows that head instances need fewer experts and the imbalance between classes gets reduced for later experts. At the test time, we collect the logits of assigned experts to make a final decision. c) RIDE outperforms SOTA methods (i.e. LFME (Xiang et al., 2020) for CIFAR100-LT, LWS (Kang et al., 2020) for ImageNet-LT and BBN (Zhou et al., 2020) for iNaturalist) on all the benchmarks.
fication loss (e.g. CE) over instance x and its label y. We call such an aggregation loss collaborative:
Lcollaborative(x, y) = L
( 1
n n∑ i=1 ψθi (fθ(x)) , y
) (8)
as it leads to correlated instead of complementary experts. To discourage correlation, we require each expert to do the job well by itself. Such an aggregation loss is essentially an individual loss, and it contributes a large portion of our performance gain in most of our experiments:
Lindividual(x, y) = n∑ i=1 L (ψθi (fθ(x)) , y) . (9)
Distribution-aware diversity loss. The individual classification loss and random initialization lead to diversified experts with a shared backbone. For long-tailed data, we add a regularization term to encourage complementary decisions from multiple experts. That is, we maximize the KL-divergence between different experts’ classification probabilities on instance x in class y over a total of c classes:
diversity loss: LD-Diversify(x, y; θi) = −1 n− 1 n∑ j=1,j 6=i DKL ( p(i)(x, y)‖p(j)(x, y) ) (10)
KL divergence: DKL(p‖q) = c∑
k=1
pk log (
pk qk
) (11)
classification by θi : p(i)(x, y) = softmax ([
ψθi (fθ(x))1 T1 ... ψθi (fθ(x))c Tc
]) . (12)
We vary the temperature T (or concentration) (Hadsell et al., 2006; Wu et al., 2018) applied to class k’s logit ψθi(fθ(x))k: For class k with nk instances, the smaller the nk, the lower the temperature Tk, the more sensitive the classification probability p is to a change in the feature ψ. Specifically,
class-wise temperature: Tk = α ( βk + 1−max
j βj
) (13)
normalized class size: βk = γ · nk
1 c ∑c s=1 ns + (1− γ). (14)
T scales linearly with the class size, ensuring βk = 1, Tk = α for a balanced set. This simple adaptation allows us to find classifiers of enough complexity for the head and enough robustness for the tail: On one hand, we need strong classifiers to handle large sample variations within head classes; on the other hand, such classifiers are prone to overfit small training data within tail classes. We adapt the temperature only after the CNN network is trained for several epochs and the feature is stabilized, similar to the training scheme for deferred reweighting (Cao et al., 2019).
Joint expert optimization. For our n experts θ1, . . . , θn with a shared backbone θ, we optimize their individual classification losses (LClassify = L, any classification loss such as CE, LDAM, and focal loss) and their mutual distribution-aware divergency losses, weighted by hyperparameter λ:
LTotal(x, y) = n∑ i=1 (LClassify(x, y; θi) + λ · LD-Diversify(x, y; θi) ) . (15)
Since these loss terms are completely symmetrical with respect to each other, the n experts learned at Stage 1 are equally good and distinctive from each other.
Routing diversified experts. To cut down the test-time computational cost that comes with multiple experts, we train a router at Stage 2 to deploy these (arbitrarily ordered) experts sequentially on an as-needed basis. Assume that the first k experts have been deployed for instance x. The router takes in the image feature and the mean logits from the first to the k-th expert, and makes a binary decision yon on whether to deploy the k + 1-th expert. If the k-th expert wrongly classifies x, but one of the rest n−k experts correctly classifies x, ideally the router should switch on, i.e., output yon = 1, and otherwise yon = 0. We construct a simple binary classifier with two fully connected layers to learn each router. Each of the n−1 routers for n experts has a shared component to reduce the feature dimensions and an individual component to make decisions.
Specifically, we normalize the image feature fθ(x) (for training stability), reduce the feature dimension (to e.g. 16 in our experiments) by a fully connected layer W1 which is shared with all routers, followed by ReLU and flattening, concatenate with the top-s ranked mean logits from the first to kth expert 1k ∑k i=1 ψθi(fθ(x)), project it to a scalar by W (k) 2 which is independent between routers, and finally apply Sigmoid function S(x) = 11+e−x to get a continuous activation value in [0,1]:
router activation: r(x) = S ( W
(k) 2
[ flatten · ReLU ( W1
fθ(x) ‖fθ(x)‖ ) 1 k ∑k i=1 ψθk(fθ(x))|top-s-components ]) . (16)
The router has a negligible size and compute, where s ranges from 30 for CIFAR100 to 50 for iNaturalist (8,142 classes). It is optimized with a weighted variant of binary CE loss:
LRouting(r(x), yon) = −ωon yon log (r(x))− (1− yon) log (1− r(x)) (17)
where ωon controls the easiness to switch on the router. We find ωon = 100 to be a good trade-off between classification accuracy and computational cost for all our experiments. At the test time, we simply threshold the activation with 0.5: If r(x) < 0.5, the classifier makes the final decision with the current collective logits, otherwise it proceeds to the next expert.
Optional self-distillation. While existing long-tail classifiers such as BBN Zhou et al. (2020) and LFME Xiang et al. (2020) have a fixed number of experts, our method could have an arbitrary number of experts to balance classification accuracy and computation cost. We can optionally apply self-distillation from a model with more (6 in our setting) experts to the same model with fewer experts for further performance gain (0.4%∼0.8% for most experiments). We choose knowledge distillation (Hinton et al., 2015) by default. Implementation details and comparisons with various distillation algorithms such as CRD (Tian et al., 2019) are investigated in Appendix Section A.2.

heading: 4 EXPERIMENTS

text:
We experiment on major long-tailed recognition benchmarks and various backbone networks.
1. CIFAR100-LT (Cao et al., 2019): CIFAR100 is sampled by class per an exponential decay across classes. We choose imbalance factor 100 and ResNet-32 (He et al., 2016) backbone.
2. ImageNet-LT (Liu et al., 2019): Multiple backbone networks are experimented on ImageNetLT, including ResNet-10, ResNet-50 and ResNeXt-50 (Xie et al., 2017). All backbone networks are trained with a batch size of 256 on 8 RTX 2080Ti GPUs for 100 epochs using SGD with an initial learning rate of 0.1 decayed by 0.1 at 60 epochs and 80 epochs. See more details and results on other backbones in Appendix.
3. iNaturalist 2018 (Van Horn et al., 2018): It is a naturally imbalanced fine-grained dataset with 8,142 categories. We use ResNet-50 as the backbone and apply the same training recipe as for ImageNet-LT except batch size 512, as in (Kang et al., 2020).
CIFAR100-LT Results. Table 1 shows that RIDE outperforms SOTA by a large margin on CIFAR100-LT. The average computational cost is even about 10% less than baseline models with two experts as in BBN. RIDE surpasses multi-expert methods, LFME (Xiang et al., 2020) and BBN (Zhou et al., 2020), by more than 5.3% and 6.5% respectively.
RIDE as a universal framework. Fig. 3 shows that RIDE consistently benefits from better loss functions and training processes. Whether the model is trained end-to-end (focal loss, CE, LDAM) or in two stages (cRT, τ -norm, cosine), RIDE delivers consistent accuracy gains. In particular, RIDE with a simple cosine classifier, which we constructed by normalizing the classifier weights and retraining them with a long-tail re-sampling strategy (similar to cRT), achieves on-par performance with the current SOTA methods. Fig. 3 also shows that two-stage methods are generally better than single-stage ones. Nevertheless, since they require an additional training stage, for simplicity, we use the single-stage LDAM as the default LClassify in RIDE throughout our remaining experiments. ImageNet-LT Results. Table 2 shows that RIDE outperforms SOTA, LWS and cRT, by more than 7.7% with ResNet-50. ResNeXt-50 is based on group convolution (Xie et al., 2017), which divides all filters into several groups and aggregates information from multiple groups. ResNeXt-50 generally performs better than ResNet-50 on multiple tasks. It provides 6.9% gain on ImageNet-LT.
iNaturalist 2018 Results. Table 3 shows that RIDE outperforms current SOTA by 6.3%. Surprisingly, RIDE obtains very similar results on many-shots, medium-shots and few-shots, ideal for long tailed recognition. Current SOTA method BBN also uses multiple experts; however, it significantly
decreases the performance on many-shots by about 23%. RIDE is remarkable at increasing the few-shot accuracy without reducing the many-shot accuracy.
Comparing with SOTAs on iNaturalist and ImageNet-LT. As illustrated in Fig. 4, our approach provides a comprehensive treatment to all the many-shot, medium-shot and few-shot classes, achieving substantial improvements to current state-of-the-art on all aspects. Compared with cRT which reduces the performance on the many-shot classes, RIDE can achieves significantly better performance on the few-shot classes without impairing the many-shot classes. Similar observations can be obtained in the comparison with the state-of-the-art method BBN (Zhou et al., 2020) on iNaturalist.
Contribution of each component of RIDE. RIDE is jointly trained with LD-Diversify and LClassify, we use LDAM for LClassify by default. Table 4 shows that the architectural change from the original ResNet-32 to the RIDE variant with 2 ∼ 4 experts contributes 2.7% ∼ 4.3% gain. Applying the individual classification loss instead of the collaborative loss brings 1.5% gain. Adding the diversity loss further improves about 0.9%. The computational cost is greatly reduced by adding the dynamic expert router. Knowledge distillation from RIDE with 6 experts obtains another 0.6% gain. All these components deliver 7.1% gain over baseline LDAM.
Impact of the number of experts. Fig. 5 shows that whether in terms of relative or absolute gains, few-shots benefit more with more experts. For example, the relative gain is 16% vs. 3.8% for few-shots and many-shots respectively. No distillation is applied in this comparison.
Figure 5: # experts vs. top-1 accuracy for each split (All, Many/Medium/Few) of CIFAR100LT. Compared with the many-shot split, which is 3.8% relatively improved by adding more experts, the few-shot split can get more benefits, that is, a relative improvement of 16.1%.
The number of experts allocated to each split. Fig. 6 shows that instances in few-shots need more experts whereas most instances in many-shots just need the first expert. That is, low confidence in tail instances often requires the model to seek a second (or a third, ...) opinion.

heading: 5 SUMMARY

text:
We take a dynamic view of training data and study long-tailed recognition with model bias and variance analysis. Existing long-tail classifiers do not reduce head-tail model bias gap enough while increasing model variance across all the classes. We propose a novel multi-expert model called RIDE to reduce model biases and variances throughout. It trains partially shared diverse distributionaware experts and routes an instance to additional experts when necessary, with computational costs comparable to a single expert. RIDE outperforms SOTA by a large margin. It is also a universal framework that works with various backbones and training schemes for consistent gains.
Acknowledgments. This work was supported, in part, by Berkeley Deep Drive, US Government fund through Etegent Technologies on Low-Shot Detection and Semi-supervised Detection, and NTU NAP and A*STAR via Industry Alignment Fund: Industry Collaboration Projects Grant.

heading: A APPENDIX

text:
A.1 DATASETS AND IMPLEMENTATIONS
We conduct experiments on three major long-tailed recognition benchmarks and different backbone networks to prove the effectiveness and universality of RIDE:
1.CIFAR100-LT (Krizhevsky, 2009; Cao et al., 2019): The original version of CIFAR-100 contains 50,000 images on training set and 10,000 images on validation set with 100 categories. The long-tailed version of CIFAR-100 follows an exponential decay in sample sizes across different categories. We conduct experiment on CIFAR100-LT with an imbalance factor of 100, i.e. the ratio between the most frequent class and the least frequent class. To make fair comparison with previous works, we follow the training recipe of (Cao et al., 2019) on CIFAR100-LT. We train the ResNet-32 (He et al., 2016) backbone network by SGD optimizer with a momentum of 0.9. CIFAR100-LT is trained for 200 epochs with standard data augmentations (He et al., 2016) and a batch size of 128 on one RTX 2080Ti GPU. The learning rate is initialized as 0.1 and decayed by 0.01 at epoch 120 and 160 respectively.
2.ImageNet-LT (Deng et al., 2009; Liu et al., 2019): ImageNet-LT is constructed by sampling a subset of ImageNet-2012 following the Pareto distribution with the power value α = 6 (Liu et al., 2019). ImageNet-LT consists of 115.8k images from 1,000 categories, with the largest and smallest categories containing 1,280 and 5 images, respectively. Multiple backbone networks are experimented on ImageNet-LT, including ResNet-10, ResNet-50 and ResNeXt-50 (Xie et al., 2017). All backbone networks are trained with a batch size of 256 on 8 RTX 2080Ti GPUs for 100 epochs using SGD with an initial learning rate of 0.1 decayed by 0.1 at 60 epochs and 80 epochs. We utilize standard data augmentations as in (He et al., 2016).
3.iNaturalist (Van Horn et al., 2018): The iNaturalist-2018 dataset is an imbalanced datasets with 437,513 training images from 8,142 classes with a balanced test set of 24,426 images. We use ResNet-50 as the backbone network and apply the same training recipe as ImageNet-LT, except that we use a batch size of 512.
A.2 ADDITIONAL EXPERIMENTS
Ablation study on distillation methods. Self-distillation step is optional but recommended if further improvements (0.4%∼0.8% for most experiments) are desired. We apply distillation from a more powerful model with more experts into a model with fewer experts. A simple way to transfer knowledge is knowledge distillation (KD) (Hinton et al., 2015), which applies KD loss (LKD = T 2DKL(~lteacher/T,~lexperti/T )) to match the distribution of logits of a teacher and a student. We found that for teacher model with more experts using smaller distillation loss factor gives better performance. We hypothesize that since we distill from the same teachers, giving large distillation factor prevents the branches from becoming as diversified as it is able to. We also explored other distillation methods, such as CRD (Tian et al., 2019), PKT (Passalis & Tefas, 2018), and SP (Tung & Mori, 2019), and compared the differences in Table 5. Although adding other methods along with KD may boost performance, the difference is small. Therefore, we opt for simplicity and use KD only unless otherwise noticed.
Detailed results for ImageNet-LT experiments. We list details of our ResNet-50 experiments in ImageNet-LT on Table 7. With 2 experts, we are able to achieve about 7% gain in accuracy with computational cost about 10% less than baseline. In contrast to previous methods that sacrifice many-shot accuracy to get few-shot accuracy, we improve on all three splits on ImageNet-LT. From 3 experts to 4 experts, we keep the same many-shot accuracy while increasing the few-shot accuracy, indicating that we are using the additional computational power to improve on the hardest part of the data rather than uniformly applying to all samples.
We also list our ResNet-10 and ResNeXt-50 experiments on Table 6 and 8, respectively, to compare against other works evaluated on these backbones. Our method also achieves lower computational cost and higher performance when compared to other methods.
Comparison with ensemble method. Since our method requires the joint decision from several experts, which raw ensembles also do, we also compare against ensembles of LDAM in Fig.7 on
Table 5: Comparison of different distillation methods. We transfer from a model based on ResNet-32 with 6 experts to a model of the same type, except with fewer experts. We use CIFAR100LT for the following comparison. No expert assignment module is used in the following experiments. Following the procedure for CRD (Tian et al., 2019), we also apply KD when we transfer from a teacher to students with other distillation methods.
Model Type #expert Distillation Method Accuracy (%) Teacher 6 49.7
Student
2 No Distillation 46.6 2 KD (Hinton et al., 2015) 47.3 2 CRD (Tian et al., 2019) 47.5 2 PKT (Passalis & Tefas, 2018) 47.2 2 SP (Tung & Mori, 2019) 47.2 3 No Distillation 47.9 3 KD (Hinton et al., 2015) 48.4 3 CRD (Tian et al., 2019) 48.5 3 PKT (Passalis & Tefas, 2018) 48.3 3 SP (Tung & Mori, 2019) 48.7 4 No Distillation 48.7 4 KD (Hinton et al., 2015) 49.3 4 CRD (Tian et al., 2019) 49.0 4 PKT (Passalis & Tefas, 2018) 48.9 4 SP (Tung & Mori, 2019) 49.0
0 100 200 300 400 500 MFlops
42
43
44
45
46
47
48
49
Ac cu
ra cy
47.0
48.0
49.1
42.0
45.0
46.5 47.0
47.7 47.9 47.9
RIDE and Ensembles of LDAM
RIDE Ensembles of LDAM
Figure 7: Comparison between our method and multiple LDAM models ensembled together. In the figure, ensembles of LDAM start from 1 ensemble (original LDAM) to 7 ensembles, and RIDE starts from 2 experts to 4 experts. Our method achieves higher accuracy with substantially less computational cost compared to ensemble method.
CIFAR100-LT. In the figure, even our method with 4 experts has less computational cost than the minimum computational cost for the ensemble of 2 LDAM models. This indicates that our model is much more efficient and powerful in terms of computational cost and accuracy than ensemble on long-tailed datasets.
t-SNE visualization. We also provide the t-SNE visualization of embedding space on CIFAR100LT as in Fig. 8. Compared with the baseline method LDAM, the feature embedding of RIDE is more compact for both the head and tail classes and better separated from the neighboring classes. This greatly reduces the difficulty for the classifier to distinguish the tail category.
What if we apply RIDE to balanced datasets? We also conducted experiments on CIFAR100 to check if our method can achieve similar performance gains on balanced datasets. However, we only obtained an improvement of about 1%, which is much smaller than the improvements observed on the CIFAR100-LT. Compared with balanced datasets, long-tailed datasets can get more benefits from RIDE.

title: LONG-TAILED RECOGNITION BY ROUTING DIVERSE DISTRIBUTION-AWARE EXPERTS