summaries: This paper presents a method for training a generative adversarial network on high resolution videos and complex datasets. They propose decomposing the discriminator in Adversarial Networks into a spatial and temporal discriminator similar to previous works, however, the temporal discriminator downsamples the input using average pooling before forwarding it through the network. In experiments, the presented method outperforms previous state-of-the-art methods in the used metrics. In addition, the videos generated from the Kinetic-600 dataset in a non-conditional setting are the most realistic looking up to date.

summaries: This paper tackles the problem of efficient video generation. The authors present a Dual-Video-Discriminator Generative Adversarial Network (DVD-GAN) composed of an image-level spatial discriminator and video-level temporal discriminator. DVD-GAN achieves state-of-the-art results when benchmarked against the FID, IS and FVD quantitative metrics. Compared to previous video generation works, DVD-GAN is the first model to present compelling qualitative results on the UCF-101 and Kinetics-600 dataset.

summaries: The paper proposes a class-conditional GAN model for video generation called DVD-GAN. The generator uses a single latent variable and uses ConvGRU modules and ResNet blocks to generate N frames. The model uses a dual discriminator, with one discriminator that discriminates individual frames, i.e. an image discriminator, and one that discriminates the whole video. This is similar to the MoCoGAN model, with the main difference being that the video discriminator operates on a smaller resolution video, thus reducing the dimensionality of the input to discriminate. The model is used to generate videos after being trained on the large-scale Kinetics-600 dataset, which contains multiple examples and has a lot of variability across videos. The main contribution of the paper is to successfully train this large GAN model on the very large-scale Kinetics dataset. The samples from the model are very visually appealing and are qualitatively  superior to any previous video prediction model.

abstractText: Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work. Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction, and achieve new state-of-the-art Fréchet Inception Distance for prediction for Kinetics600, as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600.

authors:
Martin Arjovsky
Soumith Chintala
Léon Bottou

title: video generation with progressive growing of sliced Wasserstein GANs

venue: Wasserstein GAN

authors:
Mohammad Babaeizadeh
Chelsea Finn
Dumitru Erhan
Roy H Campbell
Sergey Levine

title: Stochastic variational video prediction

authors:
Yogesh Balaji
Martin Renqiang Min
Bing Bai
Rama Chellappa
Hans Peter Graf

title: Tfgan: Improving conditioning for text-to-video synthesis

authors:
Nicolas Ballas
Li Yao
Chris Pal
Aaron Courville

title: Delving deeper into convolutional networks for learning video representations

authors:
Aayush Bansal
Shugao Ma
Deva Ramanan
Yaser Sheikh

title: Recycle-GAN: Unsupervised video retargeting

venue: In ECCV,

authors:
Shane Barratt
Rishi Sharma

title: A note on the Inception Score

authors:
Moshe Blank
Lena Gorelick
Eli Shechtman
Michal Irani
Ronen Basri

title: Actions as space-time shapes

venue: In ICCV,

authors:
Andrew Brock
Jeff Donahue
Karen Simonyan

title: Large scale GAN training for high fidelity natural image synthesis

authors:
Wonmin Byeon
Qin Wang
Rupesh Kumar Srivastava
Petros Koumoutsakos

title: Contextvp: Fully context-aware video prediction

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Joao Carreira
Andrew Zisserman

title: Quo vadis, action recognition? A new model and the Kinetics dataset

authors:
Joao Carreira
Eric Noland
Andras Banki-Horvath
Chloe Hillier
Andrew Zisserman

title: A short note about Kinetics-600

authors:
Harm De Vries
Florian Strub
Jérémie Mary
Hugo Larochelle
Olivier Pietquin
Aaron C Courville

title: Modulating early visual processing by language

venue: NeurIPS,

authors:
Emily Denton
Rob Fergus

title: Stochastic video generation with a learned prior

title: Unsupervised learning of disentangled representations from video

venue: In Advances in neural information processing systems,

authors:
Vincent Dumoulin
Jonathon Shlens
Manjunath Kudlur

title: A learned representation for artistic style

venue: In ICLR,

authors:
Frederik Ebert
Chelsea Finn
Alex X Lee
Sergey Levine

title: Self-supervised visual planning with temporal skip connections

authors:
Chelsea Finn
Ian Goodfellow
Sergey Levine

title: Unsupervised learning for physical interaction through video prediction

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Ishaan Gulrajani
Faruk Ahmed
Martin Arjovsky
Vincent Dumoulin
Aaron C Courville

title: Improved training of Wasserstein GANs

venue: In NeurIPS,

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: GANs trained by a two time-scale update rule converge to a local nash equilibrium

authors:
Jun-Ting Hsieh
Bingbin Liu
De-An Huang
Li Fei-Fei
Juan Carlos Niebles

title: Learning to decompose and disentangle representations for video prediction

venue: NeurIPS

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

authors:
Nal Kalchbrenner
Aäron van den Oord
Karen Simonyan
Ivo Danihelka
Oriol Vinyals
Alex Graves
Koray Kavukcuoglu

title: Video pixel networks

authors:
Tero Karras
Samuli Laine
Timo Aila

title: A style-based generator architecture for generative adversarial networks

authors:
Will Kay
Joao Carreira
Karen Simonyan
Brian Zhang
Chloe Hillier
Sudheendra Vijayanarasimhan
Fabio Viola
Tim Green
Trevor Back
Paul Natsev
Mustafa Suleyman
Andrew Zisserman

title: The Kinetics human action video dataset

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Durk P Kingma
Prafulla Dhariwal

title: Glow: Generative flow with invertible 1x1 convolutions

authors:
Manoj Kumar
Mohammad Babaeizadeh
Dumitru Erhan
Chelsea Finn
Sergey Levine
Laurent Dinh
Durk Kingma

title: VideoFlow: A flow-based generative model for video

authors:
Alex X Lee
Richard Zhang
Frederik Ebert
Pieter Abbeel
Chelsea Finn
Sergey Levine

title: Stochastic adversarial video prediction

authors:
Yitong Li
Martin Renqiang Min
Dinghan Shen
David Carlson
Lawrence Carin

title: Video generation from text

venue: In Thirty-Second AAAI Conference on Artificial Intelligence,

authors:
Michael Mathieu
Camille Couprie
Yann LeCun

title: Deep multi-scale video prediction beyond mean square error

authors:
Jacob Menick
Nal Kalchbrenner

title: Generating high fidelity images with subscale pixel networks and multidimensional upscaling

authors:
Takeru Miyato
Masanori Koyama

title: cGANs with projection discriminator

authors:
Takeru Miyato
Toshiki Kataoka
Masanori Koyama
Yuichi Yoshida

title: Spectral normalization for generative adversarial networks

authors:
Katsunori Ohnishi
Shohei Yamamoto
Yoshitaka Ushiku
Tatsuya Harada

title: Hierarchical video generation from orthogonal information: Optical flow and texture

venue: In AAAI,

authors:
Marc’Aurelio Ranzato
Arthur Szlam
Joan Bruna
Michaël Mathieu
Ronan Collobert
Sumit Chopra

title: Video (language) modeling: a baseline for generative models of natural videos

authors:
Ali Razavi
Aaron van den Oord
Oriol Vinyals

title: Generating diverse high-fidelity images with VQ-VAE-2

venue: arXiv preprint arXiv:1906.00446,

authors:
Masaki Saito
Shunta Saito

title: TGANv2: Efficient training of large models for video generation with multiple subsampling layers

authors:
Masaki Saito
Eiichi Matsumoto
Shunta Saito

title: Temporal generative adversarial nets with singular value clipping

authors:
Mehdi SM Sajjadi
Olivier Bachem
Mario Lucic
Olivier Bousquet
Sylvain Gelly

title: Assessing generative models via precision and recall

authors:
Tim Salimans
Ian Goodfellow
Wojciech Zaremba
Vicki Cheung
Alec Radford
Xi Chen

title: Improved techniques for training GANs

authors:
Andrew M Saxe
James L McClelland
Surya Ganguli

title: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

authors:
Khurram Soomro
Amir Roshan Zamir
Mubarak Shah

title: UCF101: A dataset of 101 human actions classes from videos in the wild

authors:
Concetto Spampinato
Sergio Palazzo
P D’Oro
Francesca Murabito
Daniela Giordano
M Shah

title: VOS-GAN: Adversarial learning of visual-temporal dynamics for unsupervised dense prediction

venue: in videos

authors:
Nitish Srivastava
Elman Mansimov
Ruslan Salakhutdinov

title: Unsupervised learning of video representations using LSTMs

venue: In ICML,

authors:
Ximeng Sun
Huijuan Xu
Kate Saenko

title: A two-stream variational adversarial network for video generation

authors:
Ilya Sutskever
James Martens
Geoffrey E Hinton

title: Generating text with recurrent neural networks

authors:
Christian Szegedy
Vincent Vanhoucke
Sergey Ioffe
Jonathon Shlens
Zbigniew Wojna

title: Rethinking the Inception architecture for computer vision

authors:
Seiya Tokui
Kenta Oono
Shohei Hido
Justin Clayton

title: Chainer: a next-generation open source framework for deep learning

venue: In Workshop on Systems for ML and Open Source Software at NeurIPS,

authors:
Du Tran
Lubomir Bourdev
Rob Fergus
Lorenzo Torresani
Manohar Paluri

title: Learning spatiotemporal features with 3D convolutional networks

authors:
Sergey Tulyakov
Ming-Yu Liu
Xiaodong Yang
Jan Kautz

title: MoCoGAN: Decomposing motion and content for video generation

authors:
Dmitry Ulyanov
Andrea Vedaldi
Victor Lempitsky

title: Instance normalization: The missing ingredient for fast stylization

authors:
Thomas Unterthiner
Sjoerd van Steenkiste
Karol Kurach
Raphael Marinier
Marcin Michalski
Sylvain Gelly

title: Towards accurate generative models of video: A new metric & challenges

authors:
Ruben Villegas
Jimei Yang
Seunghoon Hong
Xunyu Lin
Honglak Lee

title: Decomposing motion and content for natural video sequence prediction

venue: arXiv preprint arXiv:1706.08033,

authors:
Ruben Villegas
Jimei Yang
Yuliang Zou
Sungryull Sohn
Xunyu Lin
Honglak Lee

title: Learning to generate long-term future via hierarchical prediction

authors:
Carl Vondrick
Hamed Pirsiavash
Antonio Torralba

title: Generating videos with scene dynamics

authors:
Jacob Walker
Kenneth Marino
Abhinav Gupta
Martial Hebert

title: The pose knows: Video forecasting by generating pose futures

authors:
Yunbo Wang
Zhifeng Gao
Mingsheng Long
Jianmin Wang
Philip S Yu

title: Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning

venue: arXiv preprint arXiv:1804.06300,

authors:
Yunbo Wang
Lu Jiang
Ming-Hsuan Yang
Li-Jia Li
Mingsheng Long
Li Fei-Fei

title: Eidetic 3d lstm: A model for video prediction and beyond. 2018c

authors:
Dirk Weissenborn
Oscar Täckström
Jakob Uszkoreit

title: Scaling autoregressive video models

authors:
You Xie
Erik Franz
Mengyu Chu
Nils

title: Thuerey. tempoGAN: A temporally coherent, volumetric GAN for super-resolution fluid flow

authors:
Ceyuan Yang
Zhe Wang
Xinge Zhu
Chen Huang
Jianping Shi
Dahua Lin

title: Pose guided human video generation

authors:
Yipin Zhou
Zhaowen Wang
Chen Fang
Trung Bui
Tamara L Berg

title: Dance dance generation: Motion transfer for internet

venue: (Saxe et al.,

text:
1 INTRODUCTION
Modern deep generative models can produce realistic natural images when trained on high-resolution and diverse datasets (Brock et al., 2019; Karras et al., 2018; Kingma & Dhariwal, 2018; Menick & Kalchbrenner, 2019; Razavi et al., 2019). Generation of natural video is an obvious further challenge for generative modeling, but one that is plagued by increased data complexity and computational requirements. For this reason, much prior work on video generation has revolved around relatively simple datasets, or tasks where strong temporal conditioning information is available.
We focus on the tasks of video synthesis and video prediction (defined in Section 2.1), and aim to extend the strong results of generative image models to the video domain. Building upon the state-of-the-art BigGAN architecture (Brock et al., 2019), we introduce an efficient spatio-temporal decomposition of the discriminator which allows us to train on Kinetics-600 – a complex dataset of
natural videos an order of magnitude larger than other commonly used datasets. The resulting model, Dual Video Discriminator GAN (DVD-GAN), is able to generate temporally coherent, high-resolution videos of relatively high fidelity (Figure 1).
Our contributions are as follows:
• We propose DVD-GAN – a scalable generative model of natural video which produces high-quality samples at resolutions up to 256× 256 and lengths up to 48 frames. • We achieve state of the art for video synthesis on UCF-101 and prediction on Kinetics-600. • We establish class-conditional video synthesis on Kinetics-600 as a new benchmark for
generative video modeling, and report DVD-GAN results as a strong baseline.

heading: 2 BACKGROUND

heading: 2.1 VIDEO SYNTHESIS AND PREDICTION

text: The exact formulation of the video generation task can differ in the type of conditioning signal provided. At one extreme lies unconditional video synthesis where the task is to generate any video following the training distribution. Another extreme is occupied by strongly-conditioned models, including generation conditioned on another video for content transfer (Bansal et al., 2018; Zhou et al., 2019), per-frame segmentation masks (Wang et al., 2018a), or pose information (Walker et al., 2017; Villegas et al., 2017b; Yang et al., 2018). In the middle ground there are tasks which are more structured than unconditional generation, and yet are more challenging from a modeling perspective than strongly-conditional generation (which gets a lot of information about the generated video through its input). The objective of class-conditional video synthesis is to generate a video of a given category (e.g., “riding a bike”) while future video prediction is concerned with generation of continuing video given initial frames. These problems differ in several aspects, but share a common requirement of needing to generate realistic temporal dynamics, and in this work we focus on these two problems.

heading: 2.2 GENERATIVE ADVERSARIAL NETWORKS

text:
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are a class of generative models defined by a minimax game between a Discriminator D and a Generator G. The original objective was proposed by Goodfellow et al. (2014), and many improvements have since been suggested, mostly targeting improved training stability (Arjovsky et al., 2017; Zhang et al., 2018; Brock et al., 2019; Gulrajani et al., 2017; Miyato et al., 2018). We use the hinge formulation of the objective (Lim & Ye, 2017; Brock et al., 2019) which is optimized by gradient descent (ρ is the elementwise ReLU function):
D: min D E x∼data(x)
[ ρ(1−D(x)) ] + E
z∼p(z)
[ ρ(1 +D(G(z))) ] , G: max
G E
z∼p(z)
[ D(G(z)) ] .
GANs have well-known limitations including a tendency towards limited diversity in generated samples (a phenomenon known as mode collapse) and the difficulty of quantitative evaluation due
to the lack of an explicit likelihood measure over the data. Despite these downsides, GANs have produced some of the highest fidelity samples across many visual domains (Karras et al., 2018; Brock et al., 2019).

heading: 2.3 KINETICS-600

text:
Kinetics is a large dataset of 10-second high-resolution YouTube clips (Kay et al., 2017; DeepMind, 2018) originally created for the task of human action recognition. We use the second iteration of the dataset, Kinetics-600 (Carreira et al., 2018), which consists of 600 classes with at least 600 videos per class for a total of around 500,000 videos.1 Kinetics videos are diverse and unconstrained, which allows us to train large models without being concerned with the overfitting that occurs on small datasets with fixed objects interacting in specified ways (Ebert et al., 2017; Blank et al., 2005). Among prior work, the closest dataset (in terms of subject and complexity) which is consistently used is UCF-101 (Soomro et al., 2012). We focus on Kinetics-600 because of its larger size (almost 50x more videos than UCF-101) and its increased diversity (600 instead of 101 classes – not to mention increased intra-class diversity). Nevertheless for comparison with prior art we train on UCF-101 and achieve a state-of-the-art Inception Score there. Kinetics contains many artifacts expected from YouTube, including cuts (as in Figure 2d), title screens and visual effects. Except when specifically described, we choose frames with stride 2 (meaning we skip every other frame). This allows us to generate videos with more complexity without incurring higher computational cost.
To the best of our knowledge we are the first to consider generative modelling of the entirety of the Kinetics video dataset2, although a small subset of Kinetics consisting of 4,000 selected and stabilized videos (via a SIFT + RANSAC procedure) has been used in at least two prior papers (Li et al., 2018; Balaji et al., 2018). Due to the heavy pre-processing and stabilization present, as well as the sizable reduction in dataset size (two orders of magnitude) we do not consider these datasets comparable to the full Kinetics-600 dataset.

heading: 2.4 EVALUATION METRICS

text: Designing metrics for measuring the quality of generative models (GANs in particular) is an active area of research (Sajjadi et al., 2018; Barratt & Sharma, 2018). In this work we report the two most commonly used metrics, Inception Score (IS) (Salimans et al. (2016)) and Fréchet Inception Distance (FID) (Heusel et al., 2017). The standard instantiation of these metrics is intended for generative image models, and uses an Inception model (Szegedy et al., 2016) for image classification or feature extraction. For videos, we use the publicly available Inflated 3D Convnet (I3D) network trained on Kinetics-600 (Carreira & Zisserman, 2017). Our Fréchet Inception Distance is therefore very similar to the Fréchet Video Distance (FVD) (Unterthiner et al., 2018), although our implementation is different and more aligned with the original FID metric.3 More details are in Appendix A.4.

heading: 3 DUAL VIDEO DISCRIMINATOR GAN

text:
Our primary contribution is Dual Video Discriminator GAN (DVD-GAN), a generative video model of complex human actions built upon the state-of-the-art BigGAN architecture (Brock et al., 2019) while introducing scalable, video-specific generator and discriminator architectures. An overview of the DVD-GAN architecture is given in Figure 3 and a detailed description is in Appendix A.2. Unlike some of the prior work, our generator contains no explicit priors for foreground, background or motion (optical flow); instead, we rely on a high-capacity neural network to learn this in a data-driven manner. While DVD-GAN contains sequential components (RNNs), it is not autoregressive in time or in space. In other words, the pixels of each frame do not directly depend on other pixels in the video, as would be the case for auto-regressive models or models generating one frame at a time.
Generating long and high resolution videos is a heavy computational challenge: individual samples from Kinetics-600 (just 10 seconds long) contain upwards of 16 million pixels which need to be
1Kinetics is occasionally pruned and so we cannot give an exact size. 2In parallel with the concurrent work of Weissenborn et al. (2019). 3We use ‘avgpool’ features (rather than logits) by default, our I3D model is trained on Kinetics-600 (rather
than Kinetics-400), and we pre-calculate ground-truth statistics on the entire training set.
Generator
G ResNet Block
3 3
1
3 3
1 D ResNet Block
Class Conditional Batch Norm
ReLU Activation
Bilinear x2 Upsampling
Feature Concatenation
Convolution with Kernel Size k
Average Pool x2 Downsampling
k
Standard Gaussian Noise
Convolutional Gated Recurrent Unit
One-Hot Class Indicator
1
1
Frame T
Frame 0
G ResNet Block
x2
G ResNet Block
x2
G ResNet Block
x2
x Depth Times
3
3
3 Per Frame
D ResNet Block
x Depth Times
ℒS
D ResNet Block
x Depth Times
ℒT
Discriminators
Figure 3: Simplified architecture diagram of G (left) and DS /DT (right). More details in A.2.
generated in a consistent fashion. This is a particular challenge to the discriminator. For example, a generated video might contain an object which leaves the field of view and incorrectly returns with a different color. Here, the ability to determine this video is generated is only possible by comparing two different spatial locations across two (potentially distant) frames. Given a video with length T , height H , and width W , discriminators that process the entire video would have to process all H ×W × T pixels – limiting the size of the model and the size of the videos being generated.

heading: 3.1 DUAL DISCRIMINATORS

text: DVD-GAN tackles this scale problem by using two discriminators: a Spatial Discriminator DS and a Temporal Discriminator DT . DS critiques single frame content and structure by randomly sampling k full-resolution frames and judging them individually. We use k = 8 and discuss this choice in Section 4.3. DS’s final score is the sum of the per-frame scores. The temporal discriminator DT must provide G with the learning signal to generate movement (something not evaluated by DS). To make the model scalable, we apply a spatial downsampling function φ(·) to the whole video and feed its output to DT . We choose φ to be 2× 2 average pooling, and discuss alternatives in Section 4.3. This results in an architecture where the discriminators do not process the entire video’s worth of pixels, since DS processes only k × H ×W pixels and DT only T × H2 × W 2 . For a 48 frame video at 128× 128 resolution, this reduces the number of pixels to process per video from 786432 to 327680: a 58% reduction. Despite this decomposition, the discriminator objective is still able to penalize almost all inconsistencies which would be penalized by a discriminator judging the entire video. DT judges any temporal discrepancies across the entire length of the video, and DS can judge any high resolution details. The only detail the DVD-GAN discriminator objective is unable to reflect is the temporal evolution of pixels within a 2 × 2 window. We have however not noticed this affecting the generated samples in practice. DVD-GAN’s DS is similar to the per-frame discriminator DI in MoCoGAN (Tulyakov et al., 2018). However MoCoGAN’s analog of DT looks at full resolution videos, whereas DS is the only source of learning signal for high-resolution details in DVD-GAN. For this reason, DS is essential when φ is not the identity, unlike in MoCoGAN where the additional per-frame discriminator is less crucial.

heading: 3.2 RELATED WORK

text:
Generative video modeling is a widely explored problem which includes work on VAEs (Babaeizadeh et al., 2018; Denton & Fergus, 2018; Lee et al., 2018; Hsieh et al., 2018) and recurrent models (Wang et al., 2018b; Finn et al., 2016; Wang et al., 2018c; Byeon et al., 2018), auto-regressive models (Ranzato et al., 2014; Srivastava et al., 2015; Kalchbrenner et al., 2017; Weissenborn et al., 2019), normalizing flows (Kumar et al., 2019), and GANs (Mathieu et al., 2015; Vondrick et al., 2016; Saito et al., 2017; Saito & Saito, 2018). Much prior work considers decompositions which model the
texture and spatial consistency of objects separately from their temporal dynamics. One approach is to split G into foreground and background models (Vondrick et al., 2016; Spampinato et al., 2018), while another considers explicit or implicit optical flow or motion in either G or D (Saito et al., 2017; Ohnishi et al., 2018). Other methods decompose the generator (or encoder) to treat concepts like pose, content and motion separately from one another (Denton et al., 2017; Villegas et al., 2017a). Similar to DVD-GAN, MoCoGAN (Tulyakov et al., 2018) discriminates individual frames in addition to a discriminator which operates on fixed-length K-frame slices of the whole video (where K < T ). Though this potentially reduces the number of pixels to discriminate to (H ×W ) + (K ×H ×W ), Tulyakov et al. (2018) describes discriminating sliding windows, which increases the total number of pixels. Other models follow this approach by discriminating groups of frames (Xie et al., 2018; Sun et al., 2018; Balaji et al., 2018).
TGANv2 (Saito & Saito, 2018) proposes “adaptive batch reduction” for efficient training, an operation which randomly samples subsets of videos within a batch and temporal subwindows within each video. This operation is applied throughout TGANv2’s G, with heads projecting intermediate feature maps directly to pixel space before applying batch reduction, and corresponding discriminators evaluating these lower resolution intermediate outputs. An effect of this choice is that TGANv2 discriminators only evaluate full-length videos at very low resolution. We show in Figure 6 that a similar reduction in DVD-GAN’s resolution when judging full videos leads to a loss in performance. We expect further reduction (towards the resolution at which TGANv2 evaluates the entire length of video) to lead to further degradation of DVD-GAN’s quality. Furthermore, this method is not easily adapted towards models with large batch sizes divided across a number of accelerators, with only a small batch size per replica.

heading: 4 EXPERIMENTS AND ANALYSIS

text: A detailed description of our training setup is in Appendix A.3. Each DVD-GAN was trained on TPU pods (Google, 2018) using between 32 and 512 replicas with an Adam (Kingma & Ba, 2014) optimizer. Video Synthesis models are trained for around 300,000 learning steps, whilst Video Prediction models are trained for up to 1,000,000 steps. Most models took between 12 and 96 hours to train.

heading: 4.1 VIDEO SYNTHESIS

text: Our primary results concern the problem of Video Synthesis. We provide our results for the UCF-101 and Kinetics-600 datasets. With Kinetics-600 emerging as a new benchmark for generative video modelling, our results establish a strong baseline for future work.

heading: 4.1.1 KINETICS-600 RESULTS

text:
In Table 1 we show the main result of this paper: benchmarks for Class-Conditional Video Synthesis on Kinetics-600. In this regime, we train a single DVD-GAN on all classes of Kinetics-600, supplying per-sample class information to both G and D. We consider a range of resolutions and video lengths, and measure Inception Score and Fréchet Inception Distance (FID) for each (as described in Section 2.4). We further measure each model along a truncation curve, which we carry out by calculating FID and IS statistics while varying the standard deviation of the latent vectors between 0 and 1. There is no prior work with which to quantitatively compare these results (for comparative experiments see Section 4.1.2 and Section 4.2.1), but we believe these samples to show a level of fidelity not yet achieved in datasets as complex as Kinetics-600 (see samples from each row in Appendix D.1). Because all videos are resized for the I3D network (to 224× 224), it is meaningful to compare metrics across equal length videos at different resolutions. Neither IS nor FID are comparable across videos of different lengths, and should be treated as separate metrics.
Generating longer and larger videos is a more challenging modeling problem, which is conveyed by the metrics (in particular, comparing 12-frame videos across 64× 64, 128× 128 and 256× 256 resolutions). Nevertheless, DVD-GAN is able to generate plausible videos at all resolutions and with length spanning up to 4 seconds (48 frames). As can be seen in Appendix D.1, smaller videos display high quality textures, object composition and movement. At higher resolutions, generating coherent objects becomes more difficult (movement consists of a much larger number of pixels), but high-level details of the generated scenes are still extremely coherent, and textures (even complicated ones like a forest backdrop in Figure 1a) are generated well. It is further worth noting that the 48-frame models do not see more high resolution frames than the 12-frame model (due to the fixed choice of k = 8 described in Section 3.1), yet nevertheless learn to generate high resolution images.

heading: 4.1.2 VIDEO SYNTHESIS ON UCF-101

text:
We further verify our results by testing the same model on UCF-101 (Soomro et al., 2012), a smaller dataset of 13,320 videos of human actions across 101 classes that has previously been used for video synthesis and prediction (Saito et al., 2017; Saito & Saito, 2018; Tulyakov et al., 2018). Our model produces samples with an IS of 27.38, significantly outperforming the state of the art (see Table 2 for quantitative comparison and Appendix B.1 for more details).
Table 2: IS on UCF-101 without class conditioning.
Table 3: FVD on BAIR.
Method Training Set FVD (↓) Test Set FVD (↓) Video Transformer (Weissenborn et al., 2019) - 170 ± 5 DVD-GAN-FP 68.66 ± 0.78 69.15 ± 1.16 DVD-GAN 32.3 ± 0.82 31.1 ± 0.56

heading: 4.2 FUTURE VIDEO PREDICTION

text: Future Video Prediction is the problem of generating a sequence of frames which directly follow from one (or a number) of initial conditioning frames. Both this and video synthesis require G to learn to produce realistic scenes and temporal dynamics, however video prediction further requires G to analyze the conditioning frames and discover elements in the scene which will evolve over time. In this section, we use the Fréchet Video Distance exactly as Unterthiner et al. (2018): using the logits of an I3D network trained on Kinetics-400 as features. This allows for direct comparison to prior work. Our model, DVD-GAN-FP (Frame Prediction), is slightly modified to facilitate the changed problem, and details of these changes are given in Appendix A.5.

heading: 4.2.1 FRAME-CONDITIONAL KINETICS

text:
For direct comparison with concurrent work on autoregressive video models (Weissenborn et al., 2019) we consider the generation of 11 frames of Kinetics-600 at 64× 64 resolution conditioned on 5 frames, where the videos for training are not taken with any frame skipping. We show results for all these cases in Table 4. Our frame-conditional model DVD-GAN-FP outperforms the prior work on frame-conditional prediction for Kinetics. The final row labeled DVD-GAN corresponds to 16-frame class-conditional Video Synthesis samples, generated without frame conditioning and without frame skipping. The FVD of this video synthesis model is notably better.
On the one hand, we hypothesize that the synthesis model has an easier generative task: it can choose to generate (relatively) simple samples for each class, rather than be forced to continue frames taken from videos which are class outliers, or contain more complicated details. On the other hand, a certain portion of the FID/FVD metric undoubtedly comes from the distribution of objects and backgrounds present in the dataset, and so it seems that the prediction model should have a handicap in the metric by being given the ground truth distribution of backgrounds and objects with which to continue videos. The synthesis model’s improved performance on this task seems to indicate that the advantage of being able to select videos to generate is greater than the advantage of having a ground truth distribution of starting frames. This result is un-intuitive, as the frame conditional model has access to strictly more information about the data distribution it is trying to recover compared to the synthesis model (despite the fact that the two models are being trained by an identical objective). This experiment favors the synthesis model for FVD, but we highlight that other models or other metrics might produce the opposite ordering.

heading: 4.2.2 BAIR ROBOT PUSHING

text: We further test future video prediction on the single-class BAIR Robot Pushing Dataset (Ebert et al., 2017), a dataset of stationary videos of a robot arm moving around a set of changing objects. In order for direct comparison with previous results reported in Unterthiner et al. (2018), we consider generating 15 frames conditioned on a single starting frame. Like on prediction with Kinetics, we report FVD exactly as in Unterthiner et al. (2018), with ground truth statistics and conditioning frames taken from the 256-video dev set. Results are reported in Table 3. Scores are taken from Unterthiner et al. (2018). DVD-GAN-FP outperforms all prior adversarial models trained on this dataset, but performs slightly worse than Video Transformer, a concurrently developed autoregressive model Weissenborn et al. (2019). Samples from DVD-GAN-FP on BAIR are given in Figure 9.

heading: 4.3 DUAL DISCRIMINATOR INPUT

text: We analyze several choices for k (the number of frames per sample in the input to DS) and φ (the downsampling function for DT ). We expect setting φ to the identity or k = T to result in the best model, but we are interested in the maximally compressive k and φ that reduce discriminator input size (and the amount of computation), while still producing a high quality generator. For φ, we consider: 2 × 2 and 4 × 4 average pooling, the identity (no downsampling), as well as a φ which takes a random half-sized crop of the input video (as in Saito & Saito (2018)). Results can be seen in Figure 6. For each ablation, we train three identical DVD-GANs with different random initializations on 12-frame clips of Kinetics-600 at 64 × 64 resolution for 100,000 steps. We report mean and standard deviation (via the error bars) across each group for the whole training period. For k, we consider 1, 2, 8 and 10 frames. We see diminishing effect as k increases, so settle on k = 8. We note the substantially reduced IS of 4× 4 downsampling as opposed to 2× 2, and further note that taking half-sized crops (which results in the same number of pixels input to DT as 2× 2 pooling) is also notably worse.

heading: 5 CONCLUSION

text: We approached the challenging problem of modeling natural video by introducing a GAN capable of capturing the complexity of a large video dataset. We showed that on UCF-101 and frame-conditional Kinetics-600 it quantitatively achieves the new state of the art, alongside qualitatively producing video synthesis samples with high complexity and diversity. We further wish to emphasize the benefit of training generative models on large and complex video datasets, such as Kinetics-600, and envisage the strong baselines we established on this dataset with DVD-GAN will be used as a reference point by the generative modeling community moving forward. While much remains to be done before realistic videos can be consistently generated in an unconstrained setting, we believe DVD-GAN is a step in that direction.

heading: A EXPERIMENT METHODOLOGY

text:
A.1 DATASET PROCESSING
For all datasets we randomly shuffle the training set for each model replica independently. Experiments on the BAIR Robot Pushing dataset are conducted in the native resolution of 64× 64, where for UCF-101 we operate at a (downsampled) 128× 128 resolution. This is done by a bilinear resize such that the video’s smallest dimension is mapped to 128 pixels while maintaining aspect ratio (144 for UCF-101). From this we take a random 128-pixel crop along the other dimension. We use the same procedure to construct datasets of different resolutions for Kinetics-600. All three datasets contain videos with more frames than we generate, so we take a random sequence of consecutive frames from the resized output. For UCF-101, we augmented the dataset by randomly performing left-right flips with probability 0.5.
A.2 ARCHITECTURE DESCRIPTION
Our model adopts many architectural choices from Brock et al. (2019) including our nomenclature for describing network width, which is determined by the product of a channel multiplier ch with a constant for each layer in the network. The layer-wise constants for G are [8, 8, 8, 4, 2] for 64× 64 videos and [8, 8, 8, 4, 2, 1] for 128× 128. The width of the i-th layer is given by the product of ch and the i-th constant and all layers prior to the residual network in G use the initial layer’s multiplier and we refer to the product of that and ch as ch0. ch in DVD-GAN is 128 for videos with 64× 64 resolution and 96 otherwise. The corresponding ch lists for both DT and DS are [2, 4, 8, 16, 16] for 64× 64 resolution and [1, 2, 4, 8, 16, 16] for 128× 128. The input to G consists of a Gaussian latent noise z ∼ N (0, I) and a learned linear embedding e(y) of the desired class y. Both inputs are 120-dimensional vectors. G starts by computing an affine transformation of [z; e(y)] to a [4, 4, ch0]-shaped tensor (in Figure 3 this is represented as a 1× 1 convolution). [z; e(y)] is used as the input to all class-conditional Batch Normalization layers throughout G (the gray line in Figure 7).
This is then treated as the input (at each frame we would like to generate) to a Convolutional Gated Recurrent Unit (Ballas et al., 2015; Sutskever et al., 2011) whose update rule for input xt and previous output ht−1 is given by the following:
r = σ(Wr ?3 [ht−1;xt] + br)
u = σ(Wu ?3 [ht−1;xt] + bu)
c = ρ(Wc ?3 [xt; r ht−1] + bc) ht = u ht−1 + (1− u) c
In these equations σ and ρ are the elementwise sigmoid and ReLU functions respectively, the ?n operator represents a convolution with a kernel of size n× n, and the operator is an elementwise multiplication. Brackets are used to represent a feature concatenation. This RNN is unrolled once per frame. The output of this RNN is processed by two residual blocks (whose architecture is given by Figure 7). The time dimension is combined with the batch dimension here, so each frame proceeds through the blocks independently. The output of these blocks has width and height dimensions which are doubled (we skip upsampling in the first block). This is repeated a number of times, with the output of one RNN + residual group fed as the input to the next group, until the output tensors have the desired spatial dimensions. We do not reduce over the time dimension when calculating Batch Normalization statistics. This prevents the network from utilizing the Batch Normalization layers to pass information between timesteps.
The spatial discriminator DS functions almost identically to BigGAN’s discriminator, though an overview of the residual blocks is given in Figure 7 for completeness. A score is calculated for each of the uniformly sampled k frames (we default to k = 8) and the DS output is the sum over per-frame scores. The temporal discriminator DT has a similar architecture, but pre-processes the real or generated video with a 2× 2 average-pooling downsampling function φ. Furthermore, the first two residual blocks of DT are 3-D, where every convolution is replaced with a 3-D convolution with a kernel size of 3× 3× 3. The rest of the architecture follows BigGAN (Brock et al., 2019).
A.3 TRAINING DETAILS
Sampling from DVD-GAN is very efficient, as the core of the generator architecture is a feed-forward convolutional network: two 64× 64 48-frame videos can be sampled in less than 150ms on a single TPU core. The dual discriminator D is updated twice for every update of G (Heusel et al., 2017) and we use Spectral Normalization (Zhang et al., 2018) for all weight layers (approximated by the first singular value) and orthogonal initialization of weights (Saxe et al., 2013). Sampling is carried out using the exponential moving average of G’s weights, which is accumulated with decay γ = 0.9999 starting after 20,000 training steps. The model is optimized using Adam (Kingma & Ba, 2014) with batch size 512 and a learning rate of 1 ·10−4 and 5 ·10−4 for G andD respectively. Class conditioning in D (Miyato & Koyama, 2018) is projection-based whereas G relies on class-conditional Batch Normalization (Ioffe & Szegedy, 2015; De Vries et al., 2017; Dumoulin et al., 2017): equivalent to standard Batch Normalization without a learned scale and offset, followed by an elementwise affine transformation where each parameter is a function of the noise vector and class conditioning.
A.4 FID FOR KINETICS-600 SYNTHESIS
The FID we use for Synthesis on Kinetics-600 is calculated exactly as Fréchet Video Distance (Unterthiner et al., 2018) except that we use a different feature network: an I3D trained on Kinetics-600 (as opposed to the network trained on Kinetics-400 in FVD) and features from the final hidden layer instead of the logits. This metric can be implemented as a small change from the publically available FVD code (Google, 2019) by changing the name of the TF-Hub module to ’https://tfhub.dev/deepmind/i3d-kinetics-600/1’ and loading the tensor named ’RGB/inception_i3d/Logits/AvgPool3D’ from the resulting graph.
A.5 ARCHITECTURE EXTENSION TO VIDEO PREDICTION
In order to provide results on future video prediction problems we describe a simple modification to DVD-GAN to facilitate the added conditioning. A diagram of the extended model is in Figure 8.
Given C conditioning frames, our modified DVD-GAN-FP passes each frame separately through a deep residual network identical to DS . The (near) symmetric design of G and DS’s residual blocks mean that each output from a D-style residual block has a corresponding intermediate tensor in G of the same spatial resolution. After each block the resulting features for each conditioning frame are stacked in the channel dimension and passed through a 3× 3 convolution and ReLU activation. The resulting tensor is used as the initial state for the Convolutional GRU in the corresponding block in G. Note that the frame conditioning stack reduces spatial resolution while G increases resolution. Therefore the smallest features of the conditioning frames (which have been through the most layers) are input earliest in G and the larger features (which have been through less processing) are input to G towards the end. DT operates on the concatenation of the conditioning frames and the output of G, meaning that it does not receive any extra information detailing that the first C frames are special. However to reduce wasted computation we do not sample the first C frames for DS on real or generated data. This technically means that DS will never see the first few frames from real videos at full resolution, but this was not an issue in our experiments. Finally, our video prediction variant does not condition on any class information, allowing us to directly compare with prior art. This is achieved by settling the class id of all samples to 0.

heading: B FURTHER EXPERIMENTS

text:
B.1 UCF-101
UCF-101 (Soomro et al., 2012) is a dataset of 13,320 videos of human actions across 101 classes that has previously been used for video synthesis and prediction (Saito et al., 2017; Saito & Saito, 2018; Tulyakov et al., 2018). In this case, DVD-GAN is not conditioned on class labels to make our results comparable with prior work. This is achieved by setting the class labels of all input samples to 0. We report Inception Score (IS) calculated with a C3D network (Tran et al., 2015)
for quantitative comparison with prior work.4 This evaluation is performed by re-scaling the video to 128 × 128, normalizing the input features based on mean statistics of the ground truth dataset, then taking a 112 × 112 center crop and applying C3D. Our model produces samples with an IS of 27.38, significantly outperforming the state of the art (see Table 2). The DVD-GAN architecture on UCF-101 is identical to the model used for Kinetics, and is trained on 16-frame 128× 128 clips from UCF-101.
The lack of class information does hurt the performance of DVD-GAN, and training on UCF-101 with class labels leads to an improved model with an Inception Score of 32.97. This is directly comparable to Conditional TGAN Saito et al. (2017) which achieved an IS of 15.83 and is close to the IS reported for the ground truth data ( 34.49). However we note than many more recent video generation papers do not test in this regime. It is worth mentioning that our improved score is, at least partially, due to memorization of the training data. In Figure 10 we show interpolation samples from our best UCF-101 model. Like interpolations in Appendix D.2, we sample 2 latents (left and rightmost columns) and show samples from the linear interpolation in latent space along each row. Here we show 4 such interpolations (the first frame from each video). Unlike Kinetics-600 interpolations, which smoothly transition from one sample to the other, we see abrupt jumps in the latent space between highly distinct samples, and little intra-video diversity between samples in each group. It can be further seen that some generated samples highly correlate with samples from the training set.
We show this both as a failure of the Inception Score metric, the commonly reported value for classconditional video synthesis on UCF-101, but also as strong signal that UCF-101 is not a complex or diverse enough dataset to facilitate interesting video generation. Each class is relatively small, and reuse of clips from shared underlying videos means that the intra-class diversity can be restricted to just a handful of videos per class. This suggests the need for larger, more diverse and challenging datasets for generative video modelling, and we believe that Kinetics-600 provides a better benchmark for this task.

heading: C MISCELLANEOUS EXPERIMENTS

text:
Here we detail a number of modifications or miscellaneous results we experimented with which did not produce a conclusive result.
• We experimented with several variations of normalization which do not require calculating statistics over a batch of data. Group Normalization (Wu & He, 2018) performed best, almost on a par with (but worse than) Batch Normalization. We further tried Layer Normalization (Lei Ba et al., 2016), Instance Normalization (Ulyanov et al., 2016), and no normalization, but found that these significantly underperformed Batch Normalization.
• We found that removing the final Batch Normalization in G, which occurs after the ResNet and before the final convolution, caused a catastrophic failure in learning. Interestingly, just
4We use the Chainer (Tokui et al., 2015) implementation of Inception Score for C3D available at https://github.com/pfnet-research/tgan.
removing the Batch Normalization layers within G’s residual blocks still led to good (though slightly worse) generative models. In particular, variants without Batch Normalization in the residual blocks often achieve significantly higher IS (up to 110.05 for 64× 64 12 frame samples – twice normal). But these models had substantially worse FID scores (1.22 for the aforementioned model) – and produced qualitatively worse video samples.
• Early variants of DVD-GAN contained Batch Normalization which normalized over all frames of all batch elements. This gave G an extra channel to convey information across time. It took advantage of this, with the result being a model which required batch statistics in order to produce good samples. We found that the version which normalizes over timesteps independently worked just as well and without the dependence on statistics.
• Models based on the residual blocks of BigGAN-deep trained faster (in wall clock time) but slower with regards to metrics, and struggled to reach the accuracy of models based on BigGAN’s residual blocks.

heading: D GENERATED SAMPLES

text:
It is difficult to accurately convey complicated generated video through still frames. Where provided, we recommend readers view the generated videos themselves via the provided links. We refer to videos within these batches by row/column number where the video in the 0th row and column is in the top left corner.
D.1 SYNTHESIS SAMPLES
D.2 INTERPOLATION SAMPLES
We expect G to produce samples of higher quality from latents near the mean of the distribution (zero). This is the idea behind the Truncation Trick (Brock et al., 2019). Like BigGAN, we find that DVD-GAN is amenable to truncation. We also experiment with interpolations in the latent space and in the class embedding. In both cases, interpolations are evidence that G has learned a relatively smooth mapping from the latent space to real videos: this would be impossible for a network that has only memorized the training data, or which is only capable of generating a few exemplars per class. Note that while all latent vectors along an interpolation are valid (and therefore G should produce a reasonable sample), at no point during training is G asked to generate a sample halfway between two classes. Nevertheless G is able to interpolate between even very distinct classes.


summaries: A new task is suggested, similarly to FSL the test is done in an episodic manner of k-shot 5-way, but the number of samples for base classes is also limited. The model is potentially pre-trained on a large scale dataset from another domain. The suggested method is applying spatial attention according to entropy criteria (or certainty) of the original classifier (from a different domain).

summaries: This paper proposed a new realistic setting for few-shot learning that we can obtain representations from a pre-trained model trained on a large-scale dataset, but cannot access its training details. Also, there may be a large domain shift between the dataset of the pre-trained model and our dataset. For the pre-trained model, they will not only use its weights but also use it to generate a spatial attention map and help the model focuses on objects of images. Back to the standard few-shot classification problem, they will first adapt the model with base class samples and then adapt to novel classes.

summaries: The paper introduces a problem “few-shot few-shot learning” that aims to firstly transfer prior knowledge from one domain to the domain where the base training tasks reside, and then train a few-shot learning model on training tasks and apply it to novel test tasks. The two “few-shot” in the name refers to base training tasks and novel test tasks. In their algorithm, they use a model pre-trained on another dataset as the prior knowledge and fine-tune it on training tasks. During the test, they use the weighted average of samples’ representations per class as the prototype of each class, where the weight is large for samples with more discriminative prediction over pre-trained domain’s classes. Afterward, classification is reduced to finding the nearest neighbor among the class prototypes. Some experiments show that the pre-trained model can improve few-shot classification accuracy.

abstractText: Few-shot learning is often motivated by the ability of humans to learn new tasks from few examples. However, standard few-shot classification benchmarks assume that the representation is learned on a limited amount of base class data, ignoring the amount of prior knowledge that a human may have accumulated before learning new tasks. At the same time, even if a powerful representation is available, it may happen in some domain that base class data are limited or non-existent. This motivates us to study a problem where the representation is obtained from a classifier pre-trained on a large-scale dataset of a different domain, assuming no access to its training process, while the base class data are limited to few examples per class and their role is to adapt the representation to the domain at hand rather than learn from scratch. We adapt the representation in two stages, namely on the few base class data if available and on the even fewer data of new tasks. In doing so, we obtain from the pre-trained classifier a spatial attention map that allows focusing on objects and suppressing background clutter. This is important in the new problem, because when base class data are few, the network cannot learn where to focus implicitly. We also show that a pre-trained network may be easily adapted to novel classes, without meta-learning.

authors:
Stuart Andrews
Ioannis Tsochantaridis
Thomas Hofmann

title: Support vector machines for multiple-instance learning

venue: In NIPS,

authors:
Yoshua Bengio
I. Guyon
G. Dror
V. Lemaire
G. Taylor
D. Silver

title: Deep learning of representations for unsupervised and transfer learning

venue: In in Proc. of ICML,

authors:
Luca Bertinetto
João F Henriques
Philip HS Torr
Andrea Vedaldi

title: Meta-learning with differentiable closed-form solvers

venue: arXiv preprint arXiv:1805.08136,

authors:
Mathilde Caron
Piotr Bojanowski
Armand Joulin
Matthijs Douze

title: Deep clustering for unsupervised learning of visual features

venue: In ECCV,

authors:
Wei-Yu Chen
Yen-Cheng Liu
Zsolt Kira
Yu-Chiang Frank Wang
Jia-Bin Huang

title: A closer look at few-shot classification

authors:
Yunpeng Chen
Yannis Kalantidis
Jianshu Li
Shuicheng Yan
Jiashi Feng

title: Aˆ 2-nets: Double attention networks

authors:
Nikita Dvornik
Cordelia Schmid
Julien Mairal

title: Diversity with cooperation: Ensemble methods for few-shot classification

authors:
Chelsea Finn
Pieter Abbeel
Sergey Levine

title: Model-agnostic meta-learning for fast adaptation of deep networks

authors:
Yaroslav Ganin
Victor Lempitsky

title: Unsupervised domain adaptation by backpropagation

authors:
Spyros Gidaris
Nikos Komodakis

title: Dynamic few-shot visual learning without forgetting

venue: In CVPR,

authors:
Chunrui Han
Shiguang Shan
Meina Kan
Shuzhe Wu
Xilin Chen

title: Face recognition with contrastive convolution

authors:
Bharath Hariharan
Ross B. Girshick

title: Low-shot visual recognition by shrinking and hallucinating features

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Nathan Hilliard
Lawrence Phillips
Scott Howland
Artëm Yankov
Courtney D. Corley
Nathan O. Hodas

title: Few-shot learning with metric-agnostic conditional embeddings

authors:
Qibin Hou
PengTao Jiang
Yunchao Wei
Ming-Ming Cheng

title: Self-erasing network for integral object attention

authors:
Saumya Jetley
Nicholas A. Lord
Namhoon Lee
Philip H.S. Torr

title: Learn to pay attention

authors:
Gregory Koch
Richard Zemel
Ruslan Salakhutdinov

title: Siamese neural networks for one-shot image recognition

venue: In ICMLW,

authors:
Alina Kuznetsova
Hassan Rom
Neil Alldrin
Jasper Uijlings
Ivan Krasin
Jordi Pont-Tuset
Shahab Kamali
Stefan Popov
Matteo Malloci
Tom Duerig

title: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

venue: arXiv preprint arXiv:1811.00982,

authors:
Brenden Lake
Ruslan Salakhutdinov
Jason Gross
Joshua Tenenbaum

title: One shot learning of simple visual concepts

venue: In Proceedings of the Annual Meeting of the Cognitive Science Society,

authors:
Brenden M Lake
Ruslan Salakhutdinov
Joshua B Tenenbaum

title: Human-level concept learning through probabilistic program induction

authors:
Hongyang Li
David Eigen
Samuel Dodge
Matthew Zeiler
Xiaogang Wang

title: Finding task-relevant features for few-shot learning by category traversal

authors:
Wenbin Li
Lei Wang
Jinglin Xu
Jing Huo
Yang Gao
Jiebo Luo

title: Revisiting local descriptor based image-to-class measure for few-shot learning

authors:
Z. Li
D. Hoiem

title: Learning without forgetting

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Yann Lifchitz
Yannis Avrithis
Sylvaine Picard
Andrei Bursuc

title: Dense classification and implanting for few-shot learning

authors:
Ziwei Liu
Zhongqi Miao
Xiaohang Zhan
Jiayun Wang
Boqing Gong
Stella X. Yu

title: Large-scale long-tailed recognition in an open world

venue: In IEEE Conference on Computer Vision and Pattern Recognition

authors:
Dhruv Mahajan
Ross Girshick
Vignesh Ramanathan
Kaiming He
Manohar Paluri
Yixuan Li
Ashwin Bharambe
Laurens van der Maaten

title: Exploring the limits of weakly supervised pretraining

authors:
Arun Mallya
Dillon Davis
Svetlana Lazebnik

title: Piggyback: Adapting a single network to multiple tasks by learning to mask weights

authors:
Nikhil Mishra
Mostafa Rohaninejad
Xi Chen
Pieter Abbeel

title: Meta-learning with temporal convolutions

venue: arXiv preprint arXiv:1707.03141,

title: A simple neural attentive meta-learner

authors:
Hang Qi
Matthew Brown
David G Lowe

title: Low-shot learning with imprinted weights

authors:
Ilija Radosavovic
Piotr Dollar
Ross Girshick
Georgia Gkioxari
Kaiming He

title: Data distillation: Towards omni-supervised learning

authors:
Sachin Ravi
Hugo Larochelle

title: Optimization as a model for few-shot learning

authors:
Sylvestre-Alvise Rebuffi
Alexander Kolesnikov
Christoph H. Lampert

title: iCaRL: Incremental classifier and representation learning

venue: arXiv preprint arXiv:1611.07725,

authors:
Sylvestre-Alvise Rebuffi
Hakan Bilen
Andrea Vedaldi

title: Learning multiple visual domains with residual adapters

authors:
Mengye Ren
Renjie Liao
Ethan Fetaya
Richard S Zemel

title: Incremental few-shot learning with attention attractor networks

venue: arXiv preprint arXiv:1810.07218,

authors:
O Russakovsky
J Deng
H Su
J Krause
S Satheesh
S Ma
Z Huang
A Karpathy
A Khosla
M Bernstein

title: Imagenet large scale visual recognition challenge

authors:
Olga Russakovsky
Jia Deng
Hao Su
Jonathan Krause
Sanjeev Satheesh
Sean Ma
Zhiheng Huang
Andrej Karpathy
Aditya Khosla
Michael Bernstein
Alexander C. Berg
Li Fei-Fei

title: ImageNet Large Scale Visual Recognition Challenge

venue: International Journal of Computer Vision (IJCV),

authors:
Adam Santoro
Sergey Bartunov
Matthew Botvinick
Daan Wierstra
Timothy Lillicrap

title: Meta-learning with memory-augmented neural networks

authors:
Jake Snell
Kevin Swersky
Richard Zemel

title: Prototypical networks for few-shot learning

authors:
Oriol Vinyals
Charles Blundell
Tim Lillicrap
Daan Wierstra

title: Matching networks for one shot learning

authors:
C. Wah
S. Branson
P. Welinder
P. Perona
S. Belongie

title: The Caltech-UCSD Birds-200-2011 Dataset

venue: Technical Report CNS-TR-2011-001, California Institute of Technology,

authors:
Xiaolong Wang
Ross Girshick
Abhinav Gupta
Kaiming He

title: Non-local neural networks

authors:
P. Welinder
S. Branson
T. Mita
C. Wah
F. Schroff
S. Belongie
P. Perona

title: Caltech-UCSD Birds 200

venue: Technical Report CNS-TR-2010-001, California Institute of Technology,

authors:
Davis Wertheimer
Bharath Hariharan

title: Few-shot learning with localization in realistic settings

authors:
Han-Jia Ye
Hexiang Hu
De-Chuan Zhan
Fei Sha

title: Learning embedding adaptation for few-shot learning

authors:
Jaehong Yoon
Eunho Yang
Jeongtae Lee
Sung Ju Hwang

title: Lifelong learning with dynamically expandable

authors:
Jason Yosinski
Jeff Clune
Yoshua Bengio
Hod Lipson

title: How transferable are features in deep neural networks

authors:
Bolei Zhou
Aditya Khosla
Agata Lapedriza
Aude Oliva
Antonio Torralba

title: Learning deep features for discriminative localization

authors:
Bolei Zhou
Agata Lapedriza
Aditya Khosla
Aude Oliva
Antonio Torralba

title: Places: A 10 million image database for scene recognition

authors:
Yi Zhu
Yanzhao Zhou
Qixiang Ye
Qiang Qiu
Jianbin Jiao

title: Soft proposal networks for weakly supervised object localization

heading: 1 INTRODUCTION

text:
The ever improving performance of deep learning models, apart from technical progress, is largely due to the existence of large-scale datasets, fully or weakly labeled by humans (Russakovsky et al., 2015; Kuznetsova et al., 2018; Mahajan et al., 2018). At the same time, reducing the need for supervision is becoming increasingly important, e.g. by taking advantage of prior learning (Bengio et al., 2011; Mallya et al., 2018; Li & Hoiem, 2018) or exploiting unlabeled data (Caron et al., 2018; Radosavovic et al., 2018).
An extreme situation is few-shot learning (Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017; Hariharan & Girshick, 2017), where the problem is to learn novel (previously unseen) classes using only a very small labeled training set, typically not more than 10 examples per class. Here not only the annotation but even the raw data are not available. This problem is often motivated by the ability of humans to learn new tasks from few examples (Lake et al., 2011; 2015), which has given rise to meta-learning (Santoro et al., 2016; Mishra et al., 2017; Ravi & Larochelle, 2017; Bertinetto et al., 2018; Han et al., 2018), or learning to learn. In this scenario, a training set is treated as a collection of smaller sets where every class has few labeled examples.
However, there is a huge gap between the motivating example of humans learning new tasks and how the few-shot classification task is set up. On one hand, for the sake of simplicity in experiments, the base class datasets where the representation is learned from scratch, contain a few dozen or hundred classes with a few hundred examples each. This is by no means comparable to datasets available to date (Kuznetsova et al., 2018; Mahajan et al., 2018), let alone the amount of prior knowledge that a human may have accumulated before learning a new task. On the other hand, for a given domain of novel classes, e.g. bird species (Welinder et al., 2010), base class data of such size in the same domain may not exist.
In this work, we depart from the standard few-shot classification scenario in two directions. First, we allow the representation to be learned from a large-scale dataset in a domain different than the base and novel class domain. In particular, we model prior knowledge by a classifier that is pre-trained on
such a dataset, having no access to its training process. We thus maintain the difficulty of domain gap and the simplicity of experiments (by not training from scratch), while allowing a powerful representation. Second, we assume only few or zero examples per base class. Hence, the role of base classes is to adapt the representation to the domain at hand rather than learn from scratch. This scenario can be seen as few-shot version of few-shot learning.
We treat this problem as a two-stage adaptation process, first on the few base class examples if available and second on the even fewer novel class examples. Because of the limited amount of data, it is not appropriate to apply e.g. transfer learning (Bengio et al., 2011) or domain adaptation (Ganin & Lempitsky, 2014), in either of the two stages. Because the network is pre-trained, and we do not have access to its training process or data, meta-learning is not an option either. We thus resort to few steps of fine-tuning as in the meta-testing stage of Finn et al. (2017) and Ravi & Larochelle (2017).
Focusing on image classification, we then investigate the role of spatial attention in the new problem. With large base class datasets, the network can implicitly learn the relevant parts of the images where to focus. In our setup, base class data are few, so our motivation is that a spatial attention mechanism may help the classifier in focusing on objects, suppressing background clutter. We observe that although the prior classes of the pre-trained classifier may be irrelevant to a new task, uncertainty over a large number of such classes may express anything unknown like background. This is a class-agnostic property and can apply to new tasks.
In particular, given an input image, we measure the entropy-based certainty of the pre-trained classifier in its prediction on the prior classes at every spatial location and we use it to construct a spatial attention map. This map can be utilized in a variety of ways, for instance weighted spatial pooling or weighted loss per location; and at different situations like the two adaptation stages or at inference. By exploring different alternatives, we show that a pre-trained network may be easily adapted to novel classes, without meta-learning.
In the following, we provide a detailed problem formulation and related background in section 2, then we describe our spatial attention mechanism in section 3 and its use in few-shot classification in section 4. We provide experimental results in section 5, and we conclude in section 6.

heading: 2 PROBLEM, BACKGROUND, RELATED WORK AND CONTRIBUTION

text:
Few-shot classification. We are given a set of training examples X := {xi}ni=1 ⊂ X , and corresponding labels y := (yi)ni=1 ⊂ Cn where C := [c] := {1, . . . , c} is a set of base classes. The objective is to learn a representation on these data, a process that we call base training, such that we can solve new tasks. A new task comprises a set of support examples X ′ := {x′i}n ′
i=1 ⊂ X and labels y′ := (y′i) n′ i=1 ⊂ (C ′)n ′ , where n′ n and C ′ := [c′] is a set of novel classes disjoint from C. The most common setting is k′ examples per novel class, so that n′ = k′c′, referred to as c′-way, k′-shot classification. The objective now is to learn a classifier on these support data, a process that we call adaptation. This classifier should map a new query example from X to a prediction in C ′.
Few-shot few-shot classification. Few-shot classification assumes there is more data in base than novel classes, and a domain shift between the two, in the sense of no class overlap. Here we consider a modified problem where n can be small or zero, but there is another set C◦ = [c◦] of prior classes with even more data X◦ and labels y◦ with n n◦ := |X◦| and a greater domain shift to C,C ′. Again, the most common setting is k examples per base class, so that n = kc. We are using a classifier that is pre-trained on this data but we do not have direct access to either X◦,y◦, or its learning process. The objective of base training is now to adapt the representation to the domain of C,C ′ rather than learn it from scratch; but we still call it base training.
In the remaining of this section we present general background on few-shot classification that typically applies to base classes C or novel classes C ′, but may also apply to both, in which case the symbols c, C and c′, C ′ may be used interchangeably. Prior classes and the pre-trained network are only considered in the following sections.
Classifier. The classifier is a function fθ,W : X → Rc (resp. Rc ′ ) with learnable parameters θ,W , mapping a new example x ∈ X to a vector of probabilities p := fθ,W (x) over c (resp. c′) base (resp.
novel) classes. The classifier prediction is the class of maximum probability
π(p) := arg max j pj , (1)
where pj is the j-th element of p. The classifier is built on top of an embedding function φθ : X → Rr×d. Given an example x ∈ X , this function yields a r× d feature tensor φθ(x), where r represents the spatial dimensions and d the feature dimensions. For X comprising 2d images for instance, the feature is a w × h× d tensor that is the activation of the last convolutional layer, where r = w × h is the spatial resolution. The embedding is a vector in Rd in the special case r = 1.
The embedding parameters θ may be updated into θ′ in the adaptation process, in which case we have an embedding function φθ′ and classifier fθ′,W . Again θ, θ′ may be used interchangeably.
Cosine classifier. A simple form of classifier that was introduced in few-shot learning independently by Qi et al. (2018) and Gidaris & Komodakis (2018) is a parametric linear classifier that consists of a fully-connected layer without bias on top of the embedding function φθ followed by softmax. If W := (wj) c j=1 is the collection of class weights with wj ∈ Rr×d, the classifier is defined by
fθ,W (x) := σ ( τ [s(φθ(x),wj)] c j=1 ) (2)
for x ∈ X , where σ : Rm → Rm is the softmax function σ(u) := (e u1 ,...,euc )∑
j e uj for u ∈ Rc, τ ∈ R+
is a trainable scale parameter and s is cosine similarity1. We minimize the cost function
J(X,y; θ,W ) := n∑ i=1 `(fθ,W (xi), yi) (3)
over θ,W at base training, where `(p, y) := − log py for p ∈ Rc+, y ∈ C is the cross-entropy loss.
Prototypes. An alternative classifier that is more appropriate during few-shot adaptation or at testing is a prototype classifier proposed by Snell et al. (2017) and followed by Qi et al. (2018) and Gidaris & Komodakis (2018) too. If Sj := {i ∈ [n′] : y′i = j} denotes the indices of support examples labeled in class j, then the prototype of this class j is given by the average features
pj = 1 |Sj | ∑ i∈Sj φθ′(x ′ i) (4)
of those examples for j ∈ C ′. Then, denoting by P := (pj)c ′
j=1 the collection of prototypes, a query x ∈ X is classified as fθ′,P (x), as defined by (2).
Dense classifier. This is a classifier where the loss function applies densely at each spatial location of the feature tensor rather than by global pooling or flattening as implied by (2) . The classifier can be of any form but a cosine classifier (Qi et al., 2018; Gidaris & Komodakis, 2018) is studied by Lifchitz et al. (2019). In particular, the embedding φθ(x) is seen as a collection of vectors [φ(q)(x)]rq=1, where φ(q)(x) ∈ Rd is an embedding of spatial location q ∈ [r]. The classifier (2) is then generalized to fθ,W : X → Rr×c, now mapping an example to a vector of probabilities per location, defined by
fθ,W (x) := [ σ ( τ [s(φ (q) θ (x),wj)] c j=1 )]r q=1
(5)
for x ∈ X , while the class weights W are shared over locations with wj ∈ Rd. Cross-entropy applies using the same label yi for each location of example xi, generalizing the cost function (3) to
J(X,y; θ,W ) := n∑ i=1 r∑ q=1 `(f (q) θ,W (xi), yi). (6)
Related work. Prototypical networks (Snell et al., 2017) use a prototype classifier. At testing, a query is classified to the nearest prototype, while at adaptation, computing a prototype per class (4)
1For matrices u,v ∈ Rr×d, s(u,v) := 〈u,v〉 /(‖u‖ ‖v‖) with 〈·, ·〉 and ‖·‖ being the Frobenius inner product and norm respectively.
is the only learning to be done. Base training is based on meta-learning : a number of fictitious tasks called episodes are generated by randomly sampling a number of classes from C and then a number of examples in each class from X with their labels from y. These data are assumed to be support examples and queries of novel classes C ′. Labels are now available for the queries and the objective is that they are classified correctly. In imprinting (Qi et al., 2018), a cosine classifier (2) and standard cross-entropy (3) are used instead at base training. At adaptation, class prototypes P are computed (4) and imprinted in the classifier, that is, W is replaced by W ′ := (W,P ). The entire embedding function is then fine-tuned based again on (3) to make predictions on n + n′ base and novel classes, which requires the entire training data (X,y). Few-shot learning without forgetting (Gidaris & Komodakis, 2018) uses model similar to imprinting , the main difference being that only the weight parameters W of the base classes are stored and not the entire training data. At base training, a cosine classifier is trained by (3) followed by episodes. At adaptation, prototypes are adapted to W by a class attention mechanism. Model agnostic meta-learning (MAML) (Finn et al., 2017) uses a fully-connected layer as classifier. At adaptation, the entire embedding function is fine-tuned on each new task using (3) only on the novel class data, but for few steps such that the classifier does not overfit. At base training, episodes are used where the loss function mimics the iterative optimization that normally takes place at adaptation. In implanting (Lifchitz et al., 2019), a prototype classifier is trained in episodes, keeping the base embedding function fixed but attaching a parallel implant stream of convolutional layers that learns features useful for each new task.
Contribution. The problem we consider is a variant of few-shot learning that has not been studied before. It involves sequential adaptation of a given network in two stages, each comprising a limited amount of data. There are many ways of exploiting prior learning to reduce the required amount data and supervision like transfer learning (Bengio et al., 2011; Yosinski et al., 2014), domain adaptation (Ganin & Lempitsky, 2014; Rebuffi et al., 2017; Mallya et al., 2018), or incremental learning (Li & Hoiem, 2018; Rebuffi et al., 2016; Yoon et al., 2018). However, none applies to the few-shot domain where just a handful of examples are given.
Attention has been studied as a core component of several few-shot learning and meta-learning approaches (Vinyals et al., 2016; Mishra et al., 2018; Ren et al., 2018; Gidaris & Komodakis, 2018), but it always refers to examples (e.g. images) as a unit. Spatial attention on the other hand refers to neurons at different spatial locations. It is ubiquitous in several problems, for instance weakly supervised object detection (Zhou et al., 2016; Hou et al., 2018; Zhu et al., 2017) and non-local convolution (Hu et al., 2018; Wang et al., 2018; Chen et al., 2018) but has not been applied to few-shot learning until recently (Lifchitz et al., 2019; Wertheimer & Hariharan, 2019; Li et al., 2019b; Liu et al., 2019). By breaking up one example (e.g. image) to several, one at each spatial location, the supervision becomes weak, as in multiple-instance learning (Andrews et al., 2003), typical e.g. in weakly-supervised image segmentation (Jetley et al., 2018). Our spatial attention mechanism is extremely simple, based on the pre-trained network, without training on the base or novel class data.

heading: 3 SPATIAL ATTENTION FROM PRE-TRAINING

text:
We assume a pre-trained network with an embedding function φθ◦ : X → Rr×d followed by global average pooling (GAP) and a classifier that is a fully connected layer with weightsW ◦ := (w◦j ) c◦
j=1 ∈ Rd×c◦ and biases b◦ ∈ Rc◦ , denoted jointly by U◦ := (W ◦,b◦). Without re-training, we remove the last pooling layer and apply the classifier densely as in 1× 1 convolution, followed by softmax with temperature T . Then, similarly to (5), the classifier fθ◦,U◦ : X → Rr×c ◦ maps an example to a vector of probabilities per location, where classifier parameters U◦ are shared over locations:
fθ◦,U◦(x) :=
[ σ ( 1
T
( W ◦>φ
(q) θ◦ (x) + b
◦ ))]r
q=1
. (7)
We now want to apply this classifier to examples in set X (resp. X ′) of base (resp. novel) classes C (resp. C ′) in order to provide a spatial attention mechanism to embeddings obtained by parameters θ (resp. θ′). We formulate the idea on X,C, θ in this section but it applies equally to X ′, C ′, θ′. In particular, given an example x ∈ X , we use the vector of probabilities p(q) := f (q)θ◦,U◦(x) corresponding to spatial location q ∈ [r] to compute a scalar weight w(q)(x), expressing the discriminative power of the particular location q of example x.
Since x belongs to a set of classes C different than C◦, there is no ground truth to be applied to the output of the pre-trained classifier fθ◦,U◦ . However, the distribution p(q) can still be used to evaluate how discriminative the input is. We use the entropy function for this purpose, H(p) := − ∑ j pj log(pj). We map the entropy to [0, 1], measuring the certainty of the pre-trained classifier in its prediction on the prior classes C◦:
w(q)(x) := 1− H(f
(q) θ◦,U◦(x)) log c◦ (8)
for q ∈ [r], where we ignore dependence on parameters θ◦, U◦ to simplify notation, since they remain fixed. We use this as a weight for location q assuming that uncertainty over a large number of prior classes expresses anything unknown like background, which can apply to a new set of classes. We then `1-normalize the weights w(x) := [w(q)(x)]rq=1 ∈ Rr as ŵ(x) := w(x)/ ‖w(x)‖1. We call ŵ(x) the spatial attention weights of x.
The weights are applied in different ways depending on the problem. If the embedding φθ(x) is normally a vector in Rd obtained by GAP on a feature tensor Φθ(x) ∈ Rr×d as 1r ∑ q∈[r] Φ (q) θ (x) for x ∈ X , then GAP is replaced by global global weighted average pooling (GwAP):
φθ(x) := ∑ q∈[r] ŵ(q)(x)Φ (q) θ (x). (9)
for x ∈ X . We recall that this applies equally to θ′ in the case of novel classes. Figure 1 shows examples of images with spatial attention maps. Despite the fact that there has been no training involved for the estimation of attention on the particular datasets, the result can still be useful in suppressing background clutter.

heading: 4 SPATIAL ATTENTION IN FEW-SHOT CLASSIFICATION

text: Here we discuss the use of attention maps at inference on novel classes, as well as learning on novel classes. In the latter case, the weights are pre-computed for all training examples since the pre-trained network remains fixed in this process. In summary, we either replace GAP by GwAP (9) in all inputs to the embedding network, or use dense classification (5).

heading: 4.1 BASE CLASS TRAINING

text:
Starting from a pre-trained embedding network φθ◦ , we can either solve new tasks on novel classes C ′ directly, in which case θ = θ◦, or perform base class training, fine-tuning θ from θ◦. Adaptation may involve for instance fine-tuning the last layers or the entire network, applying a spatial attention
mechanism or not. Recalling that φθ◦ is still needed for weight estimation (8), the most practical setting is to fine-tune the last layers, in which case φθ shares the same backbone network with φθ◦ . Following MAML (Finn et al., 2017), we perform few gradient descent steps with low learning rate.
We use a dense classifier fθ,W : X → Rr×c (5) with class weights W . Given the few base class examples X and labels y, we learn W at the same time as fine-tuning θ by minimizing (6).

heading: 4.2 NOVEL CLASS ADAPTATION

text:
Optionally, given the few novel class support examples X ′ and labels y′, we can further adapt the embedding network, while applying our attention mechanism to the loss function. As in section 4.1, φθ′ shares the same backbone with φθ, being derived from it by fine-tuning the last layers. We perform even fewer gradient descent steps with lower learning rate
We use a prototype classifier where vector embeddings φθ′(x′) of support examples x′ ∈ X ′ are obtained by GwAP with φθ′ defined as in (9) and class prototypes P := (pj)c ′
j=1 are obtained per class by averaging embeddings of support examples as defined by (4) and updated whenever θ′ is updated. The classifier fθ′,P : X → Rc ′ is a standard cosine classifier (2) and the loss function is standard cross-entropy J(X,y; θ, P ) (3) with embedding φθ′ obtained by GwAP (9). Attention weights apply to embeddings of all inputs to the network, each time focusing on most discriminative parts. In case of no adaptation to the embedding network, we fix θ′ = θ. Computing the prototypes P (4) is then the only learning to be done and we can proceed to inference directly.

heading: 4.3 NOVEL CLASS INFERENCE

text:
At inference, as in section 4.2, we adopt a prototype classifier where vector embeddings φθ′(x′) of support examples x′ ∈ X ′ are obtained by GwAP with φθ′ defined as in (9) and class prototypes P := (pj) c′
j=1 are obtained per class by averaging embeddings of support examples as defined by (4). Then, given a query x ∈ X , we similarly obtain a vector embedding φθ′(x) by GwAP (9) and predict the class π(fθ′,P (x)) of the nearest prototype according to cosine similarity where π is given by (1) and fθ′,P by (2). We thus focus on discriminative parts of both support and query examples, suppressing background clutter.

heading: 5 EXPERIMENTS

heading: 5.1 EXPERIMENTAL SETUP

text:
Pretrained Network. We assume that we have gathered prior knowledge on unrelated visual tasks. This knowledge is modeled by a deep convolutional network, trained on a large-scale dataset. In our experiments, we choose to use a ResNet-18 (He et al., 2016) pre-trained on the Places365-Standard subset of Places365 (Zhou et al., 2017). We refer to this subset as Places. This subset contains around 1.8 million images across 365 classes. The classes are outdoor and indoor scenes. We select this dataset for its large scale, diversity of content and different nature than other popular datasets like CUB-200-2011 (see below). Images are resampled to 224×224 pixels for training. We choose ResNet-18 as the architecture of the pre-trained model as it is a powerful network that is also used in other few-shot learning studies (Chen et al., 2019; Dvornik et al., 2019), which helps in comparisons. We make no assumption on the pre-training of the network. We do not access either the pre-training process or the dataset. We rather use a publicly available converged model that has been trained with a fully-connected layer as a classifier, as assumed in section 3.
Datasets. We apply our method to two standard datasets in few-shot learning. The first is CUB-2002011 (Wah et al., 2011), referred to as CUB, originally meant for fine-grained classification, and subsequently introduced to few-shot learning by Hilliard et al. (2018). This dataset contains 11,788 images of birds across 200 classes corresponding to different species. We use the split proposed by Ye et al. (2018), where 100 classes are used as base classes and the remaining 100 as novel, out of which 50 for validation and 50 for testing. CUB images are cropped using bounding box annotations and resampled to 224×224.
The second dataset is miniImageNet (Vinyals et al., 2016), a subset of the ImageNet ILSVRC12 (Russakovsky et al., 2014) containing 100 classes with 600 images per class. Following the split from Ravi & Larochelle (2017), 64 classes are used as base classes and 36 as novel, out of which 16 for validation and 24 for testing. Originally, miniImageNet images have been down-sampled from the ImageNet resolution to 84×84. In this work, similarily to Chen et al. (2019); Dvornik et al. (2019), we resample to 224×224 instead, which is consistent with the choice of pre-trained network. Contrary to CUB, miniImageNet has some non-negligible overlap with Places. Some classes or even objects appear in both datasets. To better satisfy our assumption of domain gap, we remove the most problematic overlapping classes from miniImageNet. As detailed in Appendix A, we remove 3 base classes, 1 validation class and 2 novel classes. We refer to this pruned dataset as modified miniImageNet. For the sake of comparison and because this overlap can happen in practice, we also experiment on the original miniImageNet, as reported in Appendix B.
Evaluation protocol. To adapt to our few-shot version of few-shot learning, we randomly keep only k images per base class. We experiment with k ∈ {0, 1, 5} and k ∈ {0, 20, 50} respectively for the CUB and miniImageNet. For novel classes, we use the standard setting k′ ∈ {1, 5}. We generate a few-shot task on novel classes by selecting a support set X ′. In particular, we sample c′ classes from the validation or test set and from each class we sample k′ images. In all experiments, c′ = 5, i.e. 5-way classification. For each task we additionally sample 30 novel class images per class, to use as queries for evaluation. We report average accuracy and 95% confidence interval over 5,000 tasks for each experiment. The base class training set X contains k examples per class for each base class in C. Each experiment can be seen as a few-shot classification task on few base class examples.
Baselines. We evaluate experiments with the network being either pre-trained on Places or randomly initialized. In both cases, we report measurements for different number k of examples per base class, as well as all examples in X . In the latter case (randomly initialized), we do not use the option k = 0 because then there would be no reasonable representation to adapt or to perform inference on, given a few-shot task on novel classes. In all cases, we compare to the baselines of using no adaptation and no spatial attention. When learning from scratch, spatial attention is not applied as we do not have access to the pre-trained classifier. In the case of random initialization, and using all examples in X , we compare to Baseline++ (Chen et al., 2019) and prototypical networks (Snell et al., 2017), as reported in the benchmark by Chen et al. (2019), as well as category traversal (CTM) (Li et al., 2019a) and ensembles (Dvornik et al., 2019), all using ResNet-18. They can only be compared to our randomly initialized baseline when using base training on all data.
Implementation details. At base training, we use stochastic gradient descent with Nesterov momentum with mini-batches of size 200. At adaptation, we perform a maximum of 60 iterations over the support examples using Adam optimizer with fixed learning rate. In both cases, the learning rate,
schedule if any and number of iterations are determined on the validation set. The temperature used by (7) for the computation of the entropy is fixed per dataset, again on the validation set. In particular, we use T = 100 and T = 2.6 respectively for CUB and modified miniImageNet.

heading: 5.2 RESULTS

text:
We present results in Tables 1 and 2 respectively for CUB and modified miniImageNet.
Effect of base training. For fine-grained few-shot classification (CUB), base training is extremely important in adapting to the new domain, improving the baseline 1-shot accuracy by more than 40% with no adaptation and no spatial attention. On object classification in general (modified miniImageNet), it is less important, improving by 4.5%. It is the first time that experiments are conducted on just a small subset of the base class training set. It is interesting that 50 examples per class are bringing nearly the same improvement as all examples, i.e. hundreds per class.
Effect of (novel class) adaptation. Fine-tuning the network on k′ novel class examples per class, even fewer than k in the case of base classes, comes with the risk of over-fitting. We still show that a small further improvement is possible with a small learning rate. The improvement is more significant when k is low, in which case, more adaptation of the embedding network to the novel class domain is needed. In the extreme case of CUB dataset without base training, adaptating on only the 25 images of the 5-way 5-shot tasks brings an improvement of 8.20%.
Effect of spatial attention. Spatial attention allows focusing on the most discriminative parts of the input, which is more beneficial when fewer examples are available. The extreme case is having no base class images and only one image per novel class. In this case, most improvement comes on modified miniImageNet without base training, where spatial attention improves 5-way 5-shot classification accuracy by 1.5% after adaptation. The attention maps appear to be domain independent as they improve CUB accuracy even when no images from the bird domain have been seen (k = 0).

heading: 6 CONCLUSION

text: In this paper we address the problem of few-shot learning when even base classes images are limited in number. To address it, we use a pre-trained network on a large-scale dataset and a very simple spatial attention mechanism that does not require any training on the base or novel classes. We consider two few-shot learning datasets: CUB and miniImageNet, with different domain gaps to our prior dataset Places. Our findings indicate that even when the domain gap is large between the dataset used for pre-training and the base/novel class domains, it is still possible to get significant benefit from base class training even with a few examples, which is very important as it reduces the need for supervision. The gain from spatial attention is more pronounced in this case.

heading: A DETAILS ON REMOVING DATASET OVERLAPS

text:
We study the overlap between Places and miniImageNet by measuring, for each miniImageNet class, what is the most frequent prediction among Places classes by the pre-trained Resnet-18 classifier and what proportion of examples are classified in this class. Ranking miniImageNet classes by that proportion, we check class names on both datasets and manually inspect examples in the top-ranking classes. We chose to remove only the most clearly overlapping classes, that is, identical classes and classes of objects contained in images of a Places class. For instance, most images from the bus station - indoor class of Places contain a bus, so we chose to remove the school bus class from miniImageNet. Table A lists the classes removed from miniImageNet in this way.
B ORIGINAL miniIMAGENET RESULTS
We also run our experiments on the original miniImageNet dataset which partially overlaps our pre-training dataset Places. We use T = 2.4 for the temperature in (7). The remaining setup as for modified miniImageNet. Results are shown in Table 4.
Compared to the results of modified miniImageNet (Table 2), performances are nearly uniformly increased by 3-4% and conclusions remain the same. The increase in performance is due to having more training data, as well as putting back easily classified classes in the test dataset. Observe that, unlike CUB (cf . Table 1), CTM (Li et al., 2019a) and ensembles (Dvornik et al., 2019) perform better than our randomly initialized baseline . Our objective is not to improve the state of the art of the standard few-shot setup , but rather to study the new problem using a network pre-trained on a large-scale dataset. In this respect, our simple baseline better facilitates future research.

title: FEW-SHOT FEW-SHOT LEARNING AND THE ROLE OF SPATIAL ATTENTION