summaries: This paper introduces a new synthetic video understanding dataset, borrowing many ideas from the visual question answering dataset CLEVR. The new dataset is the first to account for all of the following fundamental aspect of videos: temporal ordering, short- and long term reasoning, and control for scene biases. Due to the inherent biases in available action recognition datasets, models that simply averages video frames do nearly as well as models that take temporal dependencies into account. In contrast, the authors show that with the proposed dataset, models without spatiotemporal reasoning largely fail.

summaries: The paper introduces CATER: a synthetically generated dataset for video understanding tasks. The dataset is an extension of CLEVR using simple motions of primitive 3D objects to produce videos of primitive actions (e.g. pick and place a cube), compositional actions (e.g. "cone is rotated during the sliding of the sphere"), and finally a 3D object localization tasks (i.e. where is the "snitch" object at the end of the video).  The construction of the dataset focuses on demonstrating that compositional action classification and long-term temporal reasoning for action understanding and localization in videos are largely unsolved problems, and that frame aggregation-based methods on real video data in prior work datasets, have found relative success not because the tasks are easy but because of dataset bias issues.

summaries: This paper proposed a new synthetic dataset (CATER) for video understanding. The authors argue that since current video datasets are heavily biased over static scenes and object structures, it is unclear whether modern spatial-temporal video models can learn to reason over temporal dimension. In order to address this problem, they design this fully observable synthetic dataset which is built upon CLEVER, along with three tasks that are customized for temporal understanding. They further conduct a variety of experiments to benchmark state-of-the-art video understanding models and show how those models more or less struggle on temporal reasoning.

abstractText: Computer vision has undergone a dramatic revolution in performance, driven in large part through deep features trained on large-scale supervised datasets. However, much of these improvements have focused on static image analysis; video understanding has seen rather modest improvements. Even though new datasets and spatiotemporal models have been proposed, simple frame-by-frame classification methods often still remain competitive. We posit that current video datasets are plagued with implicit biases over scene and object structure that can dwarf variations in temporal structure. In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved. Our dataset, named CATER, is rendered synthetically using a library of standard 3D objects, and tests the ability to recognize compositions of object movements that require long-term reasoning. In addition to being a challenging dataset, CATER also provides a plethora of diagnostic tools to analyze modern spatiotemporal video architectures by being completely observable and controllable. Using CATER, we provide insights into some of the most recent state of the art deep video architectures.

title: Maintaining knowledge about temporal intervals

title: Allen’s interval algebra. https://www.ics.uci.edu/ ̃alspaugh/ cls/shr/allen.html

authors:
Lisa Anne Hendricks
Oliver Wang
Eli Shechtman
Josef Sivic
Trevor Darrell
Bryan Russell

title: Localizing moments in video with natural language

authors:
Anton Bakhtin
Laurens van der Maaten
Justin Johnson
Laura Gustafson
Ross Girshick

title: Phyre: A new benchmark for physical reasoning

venue: NeurIPS,

authors:
Fabien Baradel
Natalia Neverova
Julien Mille
Greg Mori
Christian Wolf

title: Cophy: Counterfactual learning of physical dynamics

venue: In ICLR,

authors:
Yunlong Bian
Chuang Gan
Xiao Liu
Fu Li
Xiang Long
Yandong Li
Heng Qi
Jie Zhou
Shilei Wen
Yuanqing Lin

title: Revisiting the effectiveness of off-the-shelf temporal modeling approaches for large-scale video classification

venue: arXiv preprint arXiv:1708.03805,

title: Movement, activity and action: the role of knowledge in the perception of motion

venue: Philosophical Transactions of the Royal Society B: Biological Sciences,

authors:
David J. Butler
Jonas Wulff
Garrett B. Stanley
Michael J. Black

title: A naturalistic open source movie for optical flow evaluation

venue: In ECCV,

authors:
Joao Carreira
Andrew Zisserman

title: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

authors:
CR De Souza
A Gaidon
Y Cabon
AM Lopez Pena

title: Procedural generation of videos to train deep action recognition

authors:
Jeff Donahue
Lisa Anne Hendricks
Marcus Rohrbach
Subhashini Venugopalan
Sergio Guadarrama
Kate Saenko
Trevor Darrell

title: Long-term recurrent convolutional networks for visual recognition and description

authors:
Radu Dondera
Vlad Morariu
Larry Davis

title: Learning to detect carried objects with minimal supervision

venue: In CVPR Workshops,

authors:
Alexey Dosovitskiy
German Ros
Felipe Codevilla
Antonio Lopez
Vladlen Koltun

title: CARLA: An open urban driving simulator

venue: In CoRL,

authors:
Christoph Feichtenhofer
Axel Pinz
Richard Wildes

title: Spatiotemporal residual networks for video action recognition

venue: In NIPS,

authors:
Silvia Ferrando
Gianluca Gera
Massimo Massa
Carlo Regazzoni

title: A new method for real time abandoned object detection and owner tracking

venue: In ICIP,

authors:
Chelsea Finn
Ian Goodfellow
Sergey Levine

title: Unsupervised learning for physical interaction through video prediction

venue: In NeurIPS,

authors:
David F. Fouhey
Weicheng Kuo
Alexei A. Efros
Jitendra Malik

title: From lifestyle VLOGs to everyday interactions

authors:
Rohit Girdhar
David Fouhey
Mikel Rodriguez
Abhinav Gupta

title: Learning a predictable and generative vector representation for objects

authors:
Rohit Girdhar
Deva Ramanan
Abhinav Gupta
Josef Sivic
Bryan Russell

title: ActionVLAD: Learning spatio-temporal aggregation for action classification

authors:
Raghav Goyal
Samira Ebrahimi Kahou
Vincent Michalski
Joanna Materzynska
Susanne Westphal
Heuna Kim
Valentin Haenel
Ingo Fruend
Peter Yianilos
Moritz Mueller-Freitag
Florian Hoppe
Christian Thurau
Ingo Bax
Roland Memisevic

title: The” something something” video database for learning and evaluating visual common sense

authors:
Chunhui Gu
Chen Sun
David Ross
Carl Vondrick
Caroline Pantofaru
Yeqing Li
Sudheendra Vijayanarasimhan
George Toderici
Susanna Ricco
Rahul Sukthankar
Cordelia Schmid
Jitendra Malik

title: Ava: A video dataset of spatio-temporally localized atomic visual actions

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In CVPR,

authors:
Yun He
Soma Shirakabe
Yutaka Satoh
Hirokatsu Kataoka

title: Human action recognition without human

venue: In ECCV Workshops,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Somboon Hongeng
Ram Nevatia
Francois Bremond

title: Video-based event recognition: activity representation and probabilistic recognition methods

authors:
Gao Huang
Zhuang Liu
Laurens van der Maaten
Kilian Q Weinberger

title: Densely connected convolutional networks

authors:
Catalin Ionescu
Dragos Papava
Vlad Olaru
Cristian Sminchisescu

title: Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments

authors:
Yunseok Jang
Yale Song
Youngjae Yu
Youngjin Kim
Gunhee Kim

title: TGIF-QA: Toward spatio-temporal reasoning in visual question answering

authors:
Justin Johnson
Bharath Hariharan
Laurens van der Maaten
Li Fei-Fei
C Lawrence Zitnick
Ross Girshick

title: CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning

authors:
Andrej Karpathy
George Toderici
Sanketh Shetty
Thomas Leung
Rahul Sukthankar
Li FeiFei

title: Large-scale video classification with convolutional neural networks

authors:
Will Kay
Joao Carreira
Karen Simonyan
Brian Zhang
Chloe Hillier
Sudheendra Vijayanarasimhan
Fabio Viola
Tim Green
Trevor Back
Paul Natsev

title: The kinetics human action video dataset

venue: arXiv preprint arXiv:1705.06950,

authors:
Michał Kempka
Marek Wydmuch
Grzegorz Runc
Jakub Toczek
Wojciech Jaśkowski

title: Vizdoom: A doom-based ai research platform for visual reinforcement learning

venue: CIG,

authors:
Eric Kolve
Roozbeh Mottaghi
Daniel Gordon
Yuke Zhu
Abhinav Gupta
Ali Farhadi

title: AI2THOR: An Interactive 3D Environment for Visual AI

authors:
H. Kuehne
H. Jhuang
E. Garrote
T. Poggio
T. Serre

title: HMDB: a large video database for human motion recognition

venue: In ICCV,

title: On space-time interest points

venue: IJCV,

authors:
Gal Lavee
Ehud Rivlin
Michael Rudzsky

title: Understanding video events: A survey of methods for automatic interpretation of semantic occurrences in video

venue: Transactions on Systems, Man, and Cybernetics,

authors:
Liyuan Li
Ruijiang Luo
Ruihua Ma
Weimin Huang
Karianto Leman

title: Evaluation of an ivs system for abandoned object detection on pets

venue: PETS Workshops,

authors:
Yingwei Li
Yi Li
Nuno Vasconcelos

title: Resound: Towards action recognition without representation bias

authors:
Xiang Long
Chuang Gan
Gerard de Melo
Jiajun Wu
Xiao Liu
Shilei Wen

title: Attention clusters: Purely attention based local feature integration for video classification

authors:
Mateusz Malinowski
Carl Doersch
Adam Santoro
Peter Battaglia

title: Learning visual question answering by bootstrapping hard attention

authors:
Mateusz Malinowski
Grzegorz Swirszcz
Joao Carreira
Viorica Patraucean

title: Sideways: Depthparallel training of video models

venue: arXiv preprint arXiv:2001.06232,

authors:
Nikolaus Mayer
Eddy Ilg
Philip Häusser
Philipp Fischer
Daniel Cremers
Alexey Dosovitskiy
Thomas Brox

title: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation

authors:
Antoine Miech
Ivan Laptev
Josef Sivic

title: Learnable pooling with context gating for video classification

authors:
Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Alex Graves
Ioannis Antonoglou
Daan Wierstra
Martin Riedmiller

title: Playing atari with deep reinforcement learning

venue: In NIPS Deep Learning Workshop

authors:
George Papandreou
Tyler Zhu
Nori Kanazawa
Alexander Toshev
Jonathan Tompson
Chris Bregler
Kevin Murphy

title: Towards accurate multi-person pose estimation in the wild

authors:
Michaela Regneri
Marcus Rohrbach
Dominikus Wetzel
Stefan Thater
Bernt Schiele
Manfred Pinkal

title: Grounding action descriptions in videos

authors:
Stephan R Richter
Zeeshan Hayder
Vladlen Koltun

title: Playing for benchmarks

authors:
Ronan Riochet
Mario Ynocente Castro
Mathieu Bernard
Adam Lerer
Rob Fergus
Véronique Izard
Emmanuel Dupoux

title: Intphys: A framework and benchmark for visual intuitive physics reasoning

venue: arXiv preprint arXiv:1803.07616,

authors:
Grégory Rogez
James S Supančič
Deva Ramanan

title: First-person pose recognition using egocentric workspaces

authors:
Marcus Rohrbach
Sikandar Amin
Mykhaylo Andriluka
Bernt Schiele

title: A database for fine grained activity detection of cooking activities

authors:
Marcus Rohrbach
Michaela Regneri
Mykhaylo Andriluka
Sikandar Amin
Manfred Pinkal
Bernt Schiele

title: Script data for attribute-based recognition of composite activities

authors:
Christian Schuldt
Ivan Laptev
Barbara Caputo

title: Recognizing human actions: a local svm approach

venue: In ICPR,

authors:
Shital Shah
Debadeepta Dey
Chris Lovett
Ashish Kapoor

title: Airsim: High-fidelity visual and physical simulation for autonomous vehicles

venue: FSR,

authors:
Amir Shahroudy
Jun Liu
Tian-Tsong Ng
Gang Wang

title: NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

authors:
Vinay D. Shet
David Harwood
Larry S. Davis

title: Vidmap: video monitoring of activity with prolog

venue: In AVSS,

title: Reasoning about change: time and causation from the standpoint of artificial intelligence

venue: Technical report, Yale University,

authors:
Gunnar A. Sigurdsson
Gül Varol
Xiaolong Wang
Ali Farhadi
Ivan Laptev
Abhinav Gupta

title: Hollywood in homes: Crowdsourcing data collection for activity understanding

authors:
Gunnar A. Sigurdsson
Olga Russakovsky
Abhinav Gupta

title: What actions are needed for understanding human actions in videos

authors:
Karen Simonyan
Andrew Zisserman

title: Two-stream convolutional networks for action recognition in videos

authors:
Shuran Song
Fisher Yu
Andy Zeng
Angel X Chang
Manolis Savva
Thomas Funkhouser

title: Semantic scene completion from a single depth

authors:
Xiaomeng Song
Yucheng Shi
Xin Chen
Yahong Han

title: Explore multi-step reasoning in video question answering

venue: In ACM-MM,

authors:
Khurram Soomro
Amir Roshan Zamir
Mubarak Shah

title: UCF101: A dataset of 101 human actions classes from videos in the wild

authors:
Nitish Srivastava
Elman Mansimov
Ruslan Salakhudinov

title: Unsupervised learning of video representations using lstms

venue: In ICML,

authors:
Hao Su
Charles R. Qi
Yangyan Li
Leonidas J. Guibas

title: Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views

authors:
Christian Szegedy
Sergey Ioffe
Vincent Vanhoucke

title: Inception-v4, inception-resnet and the impact of residual connections on learning

venue: In ICLR Workshops,

authors:
Javier Sánchez Pérez
Enric Meinhardt-Llopis
Gabriele Facciolo

title: TV-L1 Optical Flow Estimation

venue: Image Processing On Line,

authors:
Makarand Tapaswi
Yukun Zhu
Rainer Stiefelhagen
Antonio Torralba
Raquel Urtasun
Sanja Fidler

title: Movieqa: Understanding stories in movies through question-answering

authors:
YingLi Tian
Rogerio Schmidt Feris
Haowei Liu
Arun Hampapur
Ming-Ting Sun

title: Robust detection of abandoned and removed objects in complex surveillance videos

venue: IEEE Transactions on Systems, Man, and Cybernetics,

authors:
Sam Toyer
Anoop Cherian
Tengda Han
Stephen Gould

title: Human pose forecasting via deep markov models

venue: In DICTA,

authors:
Du Tran
Lubomir Bourdev
Rob Fergus
Lorenzo Torresani
Manohar Paluri

title: Learning spatiotemporal features with 3d convolutional networks

authors:
Du Tran
Heng Wang
Lorenzo Torresani
Jamie Ray
Yann LeCun
Manohar Paluri

title: A closer look at spatiotemporal convolutions for action recognition

authors:
Geoffrey Vaquette
Astrid Orcesi
Laurent Lucat
Catherine Achard

title: The daily home life activity dataset: A high semantic activity dataset for online recognition

venue: In FG,

authors:
Gül Varol
Ivan Laptev
Cordelia Schmid

title: Long-term Temporal Convolutions for Action Recognition

authors:
Gül Varol
Javier Romero
Xavier Martin
Naureen Mahmood
Michael J. Black
Ivan Laptev
Cordelia Schmid

title: Learning from Synthetic Humans

authors:
Heng Wang
Cordelia Schmid

title: Action recognition with improved trajectories

authors:
Heng Wang
Alexander Kläser
Cordelia Schmid
Liu Cheng-Lin

title: Action Recognition by Dense Trajectories

authors:
Limin Wang
Yu Qiao
Xiaoou Tang

title: MoFAP: A multi-level representation for action recognition

authors:
Limin Wang
Yuanjun Xiong
Zhe Wang
Yu Qiao
Dahua Lin
Xiaoou Tang
Luc Van Gool

title: Temporal segment networks: Towards good practices for deep action recognition

authors:
Xiaolong Wang
Ross Girshick
Abhinav Gupta
Kaiming He

title: Non-local neural networks

authors:
Jiajun Wu
Tianfan Xue
Joseph J Lim
Yuandong Tian
Joshua B Tenenbaum
Antonio Torralba
William T Freeman

title: Single image 3d interpreter

authors:
Yi Wu
Yuxin Wu
Georgia Gkioxari
Yuandong Tian

title: Building generalizable agents with a realistic and rich 3d environment

authors:
Saining Xie
Chen Sun
Jonathan Huang
Zhuowen Tu
Kevin Murphy

title: Rethinking spatiotemporal feature learning for video understanding

venue: arXiv preprint arXiv:1712.04851,

title: Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

title: TSN Pretrained Models on Kinetics Dataset

authors:
Sijie Yan
Yuanjun Xiong
Dahua Lin

title: Spatial temporal graph convolutional networks for skeleton-based action recognition

venue: In AAAI,

authors:
Kexin Yi
Chuang Gan
Yunzhu Li
Pushmeet Kohli
Jiajun Wu
Antonio Torralba
Joshua B. Tenenbaum

title: Clevrer: Collision events for video representation and reasoning

authors:
Zheng Zhu
Qiang Wang
Li Bo
Wei Wu
Junjie Yan
Weiming Hu

title: Distractor-aware siamese networks for visual object tracking

authors:
C Lawrence Zitnick
Ramakrishna Vedantam
Devi Parikh

title: Adopting abstract images for semantic scene understanding

heading: 1 INTRODUCTION

text:
While deep features have revolutionized static image analysis, video descriptors have struggled to outperform classic hand-crafted descriptors (Wang & Schmid, 2013). Though recent works have shown improvements by merging image and video models by inflating 2D models to 3D (Carreira & Zisserman, 2017; Feichtenhofer et al., 2016), simpler 2D models (Wang et al., 2016b) still routinely appear among top performers in video benchmarks such as the Kinetics Challenge at CVPR’17. This raises the natural question: are videos trivially understandable by simply averaging the predictions over a sampled set of frames?
At some level, the answer must be no. Reasoning about high-level cognitive concepts such as intentions, goals, and causal relations requires reasoning over long-term temporal structure and order (Shoham, 1987; Bobick, 1997). Consider, for example, the movie clip in Fig. 1 (a), where an actor leaves the table, grabs a firearm from another room, and returns. Even though no gun is visible in the final frames, an observer can easily infer that the actor is surreptitiously carrying the gun. Needless to say, any single frame from the video seems incapable of supporting that inference, and one needs to reason over space and time in order to reach that conclusion.
As a simpler instance of the problem, consider the cup-and-balls magic routine1, or the gamblingbased shell game2, as shown in Fig. 1 (b). In these games, an operator puts a target object (ball) under one of multiple container objects (cups), and moves them about, possibly revealing the target at various times and recursively containing cups within other cups. The task at the end is to tell which of the cups is covering the ball. Even in its simplest instantiation, one can expect any human or computer system that solves this task to require the ability to model state of the world over long temporal horizons, reason about occlusion, understand the spatiotemporal implications of containment, etc. An important aspect of both our motivating examples is the adversarial nature of the task,
∗Now at Facebook AI Research 1https://en.wikipedia.org/wiki/Cups_and_balls 2https://en.wikipedia.org/wiki/Shell_game
where the operator in control is trying to make the observer fail. Needless to say, a frame by frame prediction model would be incapable of solving such tasks.
Given these motivating examples, why don’t spatiotemporal models dramatically outperform their static counterparts for video understanding? We posit that this is due to limitations of existing video benchmarks. Even though video datasets have evolved from the small regime with tens of labels (Soomro et al., 2012; Kuehne et al., 2011; Schuldt et al., 2004) to large with hundreds of labels (Sigurdsson et al., 2016; Kay et al., 2017), tasks have remained highly correlated to the scene and object context. For example, it is trivial to recognize a swimming action given a swimming pool in the background (He et al., 2016b). This is further reinforced by the fact that state of the art pose-based action recognition models (Yan et al., 2018) are outperformed by simpler frame-level models (Wang et al., 2016b) on the Kinetics (Kay et al., 2017) benchmark, with a difference of nearly 45% in accuracy! Sigurdsson et al. also found similar results for their Charades (Sigurdsson et al., 2016) benchmark, where adding ground truth object information gave the largest boosts to action recognition performance (Sigurdsson et al., 2017).
In this work, we take an alternate approach to developing a video understanding dataset. Inspired by the recent CLEVR dataset (Johnson et al., 2017) (that explores spatial reasoning in tabletop scenes) and inspired by the adversarial parlor games above (that require temporal reasoning), we introduce CATER, a diagnostic dataset for Compositional Actions and TEmporal Reasoning in dynamic tabletop scenes. We define three tasks on the dataset, each with an increasingly higher level of complexity, but set up as classification problems in order to be comparable to existing benchmarks for easy transfer of existing models and approaches. Specifically, we consider primitive action recognition, compositional action recognition, and adversarial target tracking under occlusion and containment. However, note that this does not limit the usability of our dataset to these tasks, and we provide full metadata with the rendered videos that can be used for more complex, structured prediction tasks like detection, tracking, forecasting, and so on. Our dataset does not model an operator (or hand) moving the tabletop objects, though this could be simulated as well in future variants, as in (Rogez et al., 2015).
Being synthetic, CATER can easily be scaled up in size and complexity. It also allows for detailed model diagnostics by controlling various dataset generation parameters. We use CATER to benchmark state-of-the-art video understanding models (Wang et al., 2018; 2016b; Hochreiter & Schmidhuber, 1997), and show even the best models struggle on our dataset. We also uncover some insights into the behavior of these models by changing parameters such as the temporal duration of an occlusion, the degree of camera motion, etc., which are difficult to both tune and label in real-world video data.

heading: 2 RELATED WORK

text:
Spatiotemporal networks: Video understanding for action recognition has evolved from iconic hand-designed models (Wang & Schmid, 2013; Laptev, 2005; Wang et al., 2011) to sophisticated
Dataset Size Len Task #cls TO STR LTR CSB
UCF101 (Soomro et al., 2012) 13K 7s cls 101 7 7 7 7 HMDB51 (Kuehne et al., 2011) 5K 4s cls 51 7 7 7 7
Kinetics (Kay et al., 2017) 300K 10s cls 400 7 3 7 7 AVA (Gu et al., 2018) 430 15m det 80 7 3 7 7 VLOGs (Fouhey et al., 2018) 114K 10s cls 30 7 3 7 7 DAHLIA (Vaquette et al., 2017) 51 39m det 7 3 3 3 7
TACoS (Regneri et al., 2013) 127 6m align - 3 3 3 7 DiDeMo (Anne Hendricks et al., 2017) 10K 30s align - 3 3 3 7
Charades (Sigurdsson et al., 2016) 10K 30s det 157 3 3 7 7 Something Something (Goyal et al., 2017) 108K 4s cls 174 3 3 7 3
Diving48 (Li et al., 2018) 18K 5s cls 48 3 3 7 3 Cooking (Rohrbach et al., 2012a) 44 3-41m cls 218 3 3 7 3
IKEA (Toyer et al., 2017) 101 2-4m gen - 3 3 3 3 Composite (Rohrbach et al., 2012b) 212 1-23m cls 44 3 3 3 3
TFGIF-QA (Jang et al., 2017) 72K 3s qa - 3 3 7 7 MovieQA (Tapaswi et al., 2016) 400 200s qa - 3 3 3 7
Robot Pushing (Finn et al., 2016) 57K 1s gen - 3 3 7 3 SVQA (Song et al., 2018) 12K 4s qa - 3 3 7 3 Moving MNIST (Srivastava et al., 2015) - 2s gen - 3 3 7 3 Flash MNIST (Long et al., 2018) 100K 2s cls 1024 7 3 7 3
CATER (ours) 5.5K 10s cls 36-301 3 3 3 3
spatiotemporal deep networks (Carreira & Zisserman, 2017; Simonyan & Zisserman, 2014; Girdhar et al., 2017; Wang et al., 2018; Xie et al., 2017; Tran et al., 2018; 2015). While similar developments in the image domain have lead to large improvements on tasks like classification (Szegedy et al., 2016; He et al., 2016a; Huang et al., 2017) and localization (He et al., 2017; Papandreou et al., 2017), video models have struggled to out-perform previous hand-crafted descriptors (Wang & Schmid, 2013). Even within the set of deep video architectures, models capable of temporal modeling, such as RNNs (Karpathy et al., 2014) and 3D convolutions (Tran et al., 2015; Varol et al., 2017a) have not shown significantly better performance than much simpler, per-frame prediction models, such as variants of two-stream architectures (Wang et al., 2016b; Simonyan & Zisserman, 2014). Though some recent works have shown improvements by merging image and video models by inflating 2D models to 3D (Carreira & Zisserman, 2017; Feichtenhofer et al., 2016), simple 2D models (Wang et al., 2016b) were still among the top performers in the Kinetics Challenge at CVPR’17.
Video action understanding datasets: There has been significant effort put forth to collecting video benchmarks. One line of attack employs human actors to perform scripted actions. This is typically done in controlled environments (Schuldt et al., 2004; Shahroudy et al., 2016; Ionescu et al., 2014), but recent work has pursued online crowd sourcing (Goyal et al., 2017; Sigurdsson et al., 2016). Another direction collects videos from movies and online sharing platforms. Many popular video benchmarks follow this route for diverse, in-the-wild videos, such as UCF-101 (Soomro et al., 2012), HMDB-51 (Kuehne et al., 2011) and more recently Kinetics (Kay et al., 2017) and VLOGs (Fouhey et al., 2018). As discussed earlier, such datasets struggle with the strong bias of actions with scenes and objects. Our underlying thesis is that the field of video understanding is hampered by such biases because they favor image-based baselines. While some recent work (Goyal et al., 2017; Li et al., 2018) attempts to control for this bias, it still remains a challenge for long-term reasoning tasks. One might argue that since such biases are common in the visual world, video benchmarks should reflect them. We take the view that a diverse set of benchmarks are needed to enable comprehensive diagnostics and validation of the state-of-affairs in video understanding. Table 1 shows that CATER fills a missing gap in the benchmark landscape, most notably because of its size/video length, label distribution, relative resilience to object and scene bias, and diagnostic abilities.
Synthetic data in computer vision: Our work, being synthetically generated, is also closely related to other works in using synthetic data for computer vision applications. There has been a large body of work in this direction, with the major focus on using synthetic training data for real world applications. This includes semantic scene understanding (Dosovitskiy et al., 2017; Shah et al., 2018; Richter et al., 2017), 3D scene understanding (Girdhar et al., 2016; Su et al., 2015; Wu et al., 2016; Song et al., 2017), human understanding (Varol et al., 2017b; De Souza et al., 2017), optical flow (Butler et al., 2012; Mayer et al., 2016) and navigation, RL or embodied learning (Wu et al., 2018; Kolve et al., 2017; Kempka et al., 2016; Mnih et al., 2013). Our work, on the other hand, attempts to develop a benchmark for video based action understanding. Similar attempts have been made for scene understanding through abstract scenes (Zitnick et al., 2016), with more recently
focusing on building a complex reasoning benchmark, CLEVR (Johnson et al., 2017). In the video domain, benchmarks such as Flash-MNIST (Long et al., 2018), Moving MNIST (Srivastava et al., 2015) and SVQA (Song et al., 2018) have been proposed. Concurrent to us, CLEVRER (Yi et al., 2020), PHYRE (Bakhtin et al., 2019), COPHY (Baradel et al., 2020) and IntPhys (Riochet et al., 2018) benchmarks have been proposed with a focus on causal physical reasoning through QA, RL, prediction and ranking interfaces respectively. On the other hand, CATER focuses on spatiotemporal video reasoning tasks building upon CLEVR, with a simple classification interface, making it easily amenable for existing video understanding systems.
Object tracking: Detecting and tracking objects has typically been used as an initial representation for long-term video and activity understanding (Shet et al., 2005; Hongeng et al., 2004; Lavee et al., 2009). Extensions include adversarial tracking, where the objects are designed to be hidden from plain view. It has typically been used for tasks such as determining if humans are carrying an object (Dondera et al., 2013; Ferrando et al., 2006) or abandoned / exchanging objects (Tian et al., 2011; Li et al., 2006). We embrace this direction of work and include state-of-the-art deep trackers (Zhu et al., 2018) in our benchmark evaluation.

heading: 3 THE CATER DATASET

text:
CATER provides a video understanding dataset that requires long term temporal reasoning to be solved. Additionally, it provides diagnostic tools that can evaluate video models in specific scenarios, such as with or without camera motion, with varying number of objects and so on. This control over the dataset parameters is achieved by synthetically rendering the data. These videos come with a ground truth structure that can be used to design various different video understanding tasks, including but not limited to object localization and spatiotemporal action composition. Unlike existing video understanding benchmarks, this dataset is free of object or scene bias, as the same set of simple objects are used to render the videos. Fig. 2 describes the dataset and the associated tasks. We provide sample videos from the dataset in the supplementary video.
Objects: The CATER universe is built upon CLEVR (Johnson et al., 2017), inheriting most of the standard object shapes, sizes, colors and materials present in it. This includes three object shapes (cube, sphere, cylinder), in three sizes (small, medium, large), two materials (shiny metal and matte rubber) and eight colors, as well as a large “table” plane on which all objects are placed. In addition to these objects, we add two new object shapes: inverted cones and a special object called a ‘snitch’. Cones also come in the same set of sizes, materials and colors. The ‘snitch’ is a special object shaped like three intertwined toruses in metallic gold color.
Actions: We define four atomic actions: ‘rotate’, ‘pick-place’, ‘slide’ and ‘contain’; a subset of which is afforded by each object. The ‘rotate’ action means that the object rotates by 90°about its Y (or horizontal) axis, and is afforded by cubes, cylinders and the snitch. The ‘pick-place’ action means the object is picked up into the air along the Y axis, moved to a new position, and placed down. This is afforded by all objects. The ‘slide’ action means the object is moved to a new location by sliding along the bottom surface, and is also afforded by all objects. Finally, ‘contain’ is a special operation, only afforded by the cones, in which a cone is pick-placed on top of another object, which may be a sphere, a snitch or even a smaller cone. This allows for recursive containment, as a cone can contain a smaller cone that contains another object. Once a cone ‘contains’ an object, it is constrained to only ‘slide’ actions and effectively slides all objects contained within the cone. This holds until the top-most cone is pick-placed to another location, effectively ending the containment for that top-most cone.
Animation process: We start with an initial setup similar to CLEVR. A random number (N ) of objects with random parameters are spawned at random locations at the beginning of the video. They exist on a 6× 6 portion of a 2D plane with the global origin in the center. In addition to the random objects, we ensure that every video has a snitch and a cone. For the purposes of this work, we render 300-frame 320x240px videos, at 24 FPS, making it comparable to standard benchmarks (Soomro et al., 2012; Kuehne et al., 2011; Kay et al., 2017). We split the video into 30-frame slots, and each action is contained within these slots. At the beginning of each slot, we iterate through up to K objects in a random order and attempt to add an action afforded by that object one by one without colliding with another object. As we describe later, we use K = 2 for our initial tasks and K = N for the final task. For each action, we pick a random start and end time from within the 30-frame slot.
To further add to the diagnostic ability of this dataset, we render an additional set of videos with camera motion, with all other aspects of the data similarly distributed as the static camera case. For this, the camera is always kept pointed towards the global origin, and moved randomly between a predefined set of 3D coordinates. These coordinates include X and Y ∈ {−10, 10} and Z ∈ {8, 10, 12}. Every 30 frames, we randomly pick a new location from the Cartesian product of X,Y, Z, and move the camera to that location over the next 30 frames. However, we do constrain the camera to not change both X and Y coordinates at the same time, as that causes a jarring viewpoint shift as the camera passes over the (0, 0, Z) point. Also, we ensure all the camera motion videos start from the same viewpoint, to make it easy to register the axes locations for localization task.
Spatiotemporal compositions: We wish to label our animations with the atomic actions present, as well as their compositions. Atomic actions have a well-defined spatiotemporal footprint, and so we can define composites using spatial relations (“a cylinder is rotating behind a sliding red ball”), similar to CLEVR. Unique to CATER is the ability to designate temporal relationships (“a cylinder rotates before a ball is picked-and-placed”). Because atomic actions occupy a well-defined temporal extent, we need temporal logic that reasons about relations between intervals rather than instantaneous events. While the latter can be dealt with timestamps, the former can be described with Allen’s interval algebra with thirteen basic relations (Figure 3) along with composition operations. For simplicity, we group those into three broad relations. However, our dataset contains examples of all such interval relations and can be used to explore fine-grained temporal relationships.

heading: 3.1 TASKS DEFINED ON THE DATASET

text:
Given this CATER universe with videos, ground truth objects and their actions at any time point, we can define arbitrarily complex tasks for a video understanding system. Our choice of tasks is informed by two of the main goals of video understanding: 1) Recognizing the states of the actor, including spatiotemporal compositions of those atomic actions. For example, a spatiotemporal composition of atomic human body movements can be described as an exercise or dance routine. And 2) Recognizing the effect of those actions on the state of the world. For example, an action involving picking and placing a cup would change the position of the cup and any constituent objects contained within it, and understanding this change in the world state would implicitly require understanding the action itself.
Given these two goals, we define three tasks on CATER. Each has progressively higher complexity, and tests for a higher level reasoning ability. To be consistent with existing popular benchmarks (Soomro et al., 2012; Kuehne et al., 2011; Kay et al., 2017; Sigurdsson et al., 2016), we stick to standard single or multi-label classification setup, with standard evaluation metrics, as described next. For each of these tasks, we start by rendering 5500 total videos, to be comparable in size with existing popular benchmarks (Kuehne et al., 2011). Since tasks 1 and 2 (defined next) explicitly require recognizing individual actions, we use K = 2 for the videos rendered to keep the number of actions happening in any given video small. For task 3, we set K = N as the task is to recognize the end effect of actions, and not necessarily the actions themselves. We split the data randomly in 70:30 ratio into a training and test set. We similarly render a same size dataset with camera motion, and define tasks and splits in the same way as for the static camera. With the code release we also provide a further split of train set into a validation set (80:20). While we focus on the following tasks in this paper, note that the data is amenable to many other tasks, for instance (Malinowski et al., 2020) uses CATER for video reconstruction.
Task 1: Atomic action recognition. This first task on CATER is primarily designed as a simple debugging task, which should be easy for contemporary models to solve. Given the combinations of object shapes and actions afforded by them, we define 14 classes such as ‘slide(cone)’, ‘rotate(cube)’ and so on. Since each video can have multiple actions, we define it as a multi-label classification problem. The task is to produce 14 probability values, denoting the likelihood of that action happening in the video. The performance is evaluated using average precision per-class. Final dataset-level performance is computed by mean over all classes, to get mean average precision (mAP). This is a popular metric used in other multi-label action classification datasets (Sigurdsson et al., 2016; Gu et al., 2018).
Task 2: Compositional action recognition. While recognizing individual objects and motions is important, it is clearly not enough. Real world actions tend to be composite in nature, and humans have no difficulty recognizing them in whole or in parts. To that end, we construct a compositional action recognition task through spatiotemporal composition of the basic actions used in Task 1. For simplicity, we limit composites to pairs of 14 atomic actions, where the temporal relation is grouped into broad categories of ‘before’, ‘during’ and ‘after’ as shown in Figure 3. Combining all possible atomic actions with the three possible relations, we get a total of 14 × 14 × 3 = 588 classes, and removing duplicates (such as ‘X after Y’ is a duplicate of ‘Y before X’), leaves 301 classes. Similar to task 1, multiple compositions can be active in any given video, so we set it up as a multi-label classification problem, evaluated using mAP. If certain compositions never occur in the dataset, those are ignored for the final evaluation.
Task 3: Snitch localization. The final, and the flagship task in CATER, tests models’ ability to recognize the effect of actions on the environment. Just as in the case of cup-and-ball trick, the ability of a model to recognize location of objects after some activity can be thought of as an implicit evaluation of its ability to understand the activity itself. The task now is to predict the location of the special object introduced above, the Snitch. While it may seem trivial to localize it from the last frame, it may not always be possible to do that due to occlusions and recursive containments. The snitch can be contained by other objects (cones), which can further be contained by other larger cones. All objects move together until ‘uncontained’, so the final location of the snitch would require long range reasoning about these interactions. For simplicity, we pose this as a classification problem by quantizing the 6 × 6 grid into 36 cells and asking which cell the snitch is in, at the end of the video. We ablate the grid size in experiments. Since the snitch can only be at a single location at the end of the video, we setup the problem as a single label classification, and evaluate it using standard
percentage accuracy metrics such as top-1 and top-5 accuracy. However, one issue with this metric is that is would penalize predictions where the snitch is slightly over the cell boundaries. While the top-5 metric is somewhat robust to this issue, we also report mean L1 distance of predicted grid cell from the ground truth, as a metric that is congnizant of the grid structure in this task. Hence, it would penalize confusion between adjacent cells less than those between distant cells. The data is also amenable to a purely regression-style evaluation, though we leave that to future work.

heading: 4 EXPERIMENTS

text:
We now experiment with CATER using recently introduced state of the art video understanding and temporal reasoning models (Carreira & Zisserman, 2017; Wang et al., 2018; 2016b; Hochreiter & Schmidhuber, 1997). I3D (Carreira & Zisserman, 2017), called R3D when implemented using a ResNet (He et al., 2016a) in (Wang et al., 2018), brings the best of image models to video domain by inflating it into 3D for spatiotemporal feature learning. Non-local networks (Wang et al., 2018) further build upon that to add a spatiotemporal interaction layer that gives strong improvements and out-performs many multi-stream architectures (that use audio, flow etc) on Kinetics and Charades benchmarks. For our main task, snitch localization, we also experiment with a 2D-conv based approach, Temporal Segment Networks (TSN) (Wang et al., 2016b), which another top performing method on standard benchmarks (Kay et al., 2017). This approaches uses both RGB and flow modalities. All these architectures learn a model for individual frames or short clips, and at test time aggregate the predictions by averaging over those clips.
While simple averaging works well enough on most recent datasets (Kay et al., 2017; Soomro et al., 2012; Kuehne et al., 2011), it clearly loses all temporal information and may not be well suited to our set of tasks. Hence, we also experiment with a learned aggregation strategy: specifically using an LSTM (Hochreiter & Schmidhuber, 1997) for aggregation, which is the tool of choice for temporal modelling in various domains including language and audio. We use a common LSTM implementation for aggregating either (Wang et al., 2016b) or (Wang et al., 2018) that operates on the last layer features (before logits). We extract these features for subclips from train and test videos, and train a 2-layer LSTM with 512 hidden units in each layer on the train subclips. The LSTM produces an output at each clip it sees, and we enforce a classification loss at the end, once the model has seen all the clips. At test time we take the prediction from the last clip as the aggregated prediction. We report the LSTM performance averaged over three runs to control for random variation. It is worth noting that LSTMs have been previously used for action recognition in videos (Donahue et al., 2015; Karpathy et al., 2014), however with only marginal success over simple average pooling. As we show later, LSTMs actually perform significantly better on CATER, indicating the importance of temporal reasoning.
For task 3, we also experiment with a state-of-the-art visual tracking method (Zhu et al., 2018). We start by using the GT information of the starting position of snitch, and project it to screen coordinates using the render camera parameters. We defined a fixed size box around it to initialize the tracker, and run it until the end of the video. At the last frame, we project the center point of the tracked box to the 3D plane (and eventually, the class label) by using a homography transformation between the image and the 3D plane. This provides a more traditional, symbolic reasoning baseline for our dataset, and as we show in results, is also not enough to solve the task. Finally, we do note that many other video models have been proposed in literature involving 2.5D convolutions (Tran et al., 2018; Xie et al., 2017), VLAD-style aggregation (Girdhar et al., 2017; Miech et al., 2017) and other multi-modal architectures (Wang et al., 2016a; Bian et al., 2017). We focus on the most popular and best performing models, and leave a more comprehensive study to future work. A random baseline is also provided for all tasks, computed as the average performance of random scores passed into the evaluation functions. Implementation details for all baselines are provided in the supplementary and code will be released.
Task 1: Atomic action recognition: Table 2 (a) shows the performance of R3D with and without the non-local (NL) blocks, using different number of frames in the clips. We use a fixed sampling rate of 8, but experiment with different clip sizes. Adding more frames helps significantly in this case. Given the ease of the task, R3D obtains fairly strong performance for static camera, but not so much for moving camera, suggesting potential future work in building models agnostic to camera motion.
Table 2: Performance on the (a) 14-way atomic actions recognition, (b) 301-way compositional action recognition, and (c) 36-way localization task, for different methods.
(a) Task 1 (Atomic)
Camera Model NL #frames mAP
- Random - - 56.2
Static R3D 8 89.0 Static R3D X 8 88.8 Static R3D 32 98.8 Static R3D X 32 98.9
Moving R3D 8 82.4 Moving R3D X 8 82.7 Moving R3D 32 90.5 Moving R3D X 32 90.2
(b) Task 2 (Compositional)
Camera Model NL #frames mAP Avg LSTM
- Random - - 19.5 19.5
Static R3D 8 39.5 52.1 Static R3D 32 44.2 53.4 Static R3D X 32 45.9 53.1 Static R3D 64 43.7 43.5
Moving R3D 32 40.9 43.2 Moving R3D X 32 41.1 43.5
(c) Task 3 (Localization)
Camera Model #frames SR Avg LSTM Top 1 Top 5 L1 Top 1 Top 5 L1
- Random - - 2.8 13.8 3.9 2.8 13.8 3.9
Static Tracking - - 33.9 - 2.4 33.9 - 2.4
Static TSN (RGB) 1 - 7.4 27.0 3.9 15.3 50.0 3.0 Static TSN (RGB) 3 - 14.1 38.5 3.2 25.6 67.2 2.6 Static TSN (Flow) 1 - 6.2 21.7 4.4 7.3 26.9 4.1 Static TSN (Flow) 3 - 9.6 32.2 3.7 14.0 43.5 3.2
Static R3D 8 8 24.0 54.8 2.7 34.2 64.6 1.8 Static R3D 16 8 26.2 56.3 2.6 24.2 48.9 2.5 Static R3D 32 8 28.8 68.7 2.6 45.5 67.7 1.6 Static R3D 64 8 57.4 78.4 1.4 60.2 81.8 1.2
Static R3D + NL 32 8 26.7 68.9 2.6 46.2 69.9 1.5
Moving R3D 32 8 23.4 61.1 2.5 28.6 63.3 1.7 Moving R3D + NL 32 8 27.5 68.8 2.4 38.6 70.2 1.5
Models Kinetics UCF-101 HMDB-51 CATER
1 frame (RGB) (Donahue et al., 2015) - 67.4 - 7.4 LSTM (RGB) (Donahue et al., 2015) - 68.2 - 15.3
TSN (RGB) (Wang et al., 2016b) 72.5 93.2 51.0 14.1 TSN (Flow) (Wang et al., 2016b) 62.8 95.3 64.2 9.6 2S I3D (Carreira & Zisserman, 2017) 75.7 98.0 80.7 - 2S R(2+1)D (Tran et al., 2018) 75.4 97.3 78.7 - R3D(+NL) (Wang et al., 2018) 77.7 - - 57.4
Table 3: Long term reasoning. Comparing the best reported performance of standard models on existing datasets and CATER (task 3). Unlike previous benchmarks, (1) temporal modeling using LSTM helps and (2) local temporal cues (flow) are not effective by itself on CATER. 2S here refers to ‘Two Stream’. TSN performance from (Xiong, 2017; 2016).
Task 2: Compositional action recognition: Next we experiment with the compositional action recognition task. The training and testing is done in the same way as Task 1, except this predicts confidence over 301 classes. As evident from Table 2 (b), this task is harder for the existing models, presumably as recognizing objects and simple motions would no longer solve it, and models need to reason about spatiotemporal compositions as well. It is interesting to note that non-local blocks now add to the final performance, which was not the case for Task 1, suggesting modeling spatiotemporal relations is more useful for this task. LSTM aggregation also helps quite a bit as the model can learn to reason about long-range temporal compositions. As expected, moving camera makes the problem harder.
Task 3: Snitch localization: Finally we turn to the localization task. Since this is setup as a single label classification, we use softmax cross entropy loss to train and classification accuracy for evaluation. For tracking, no training is required as we use the pre-trained model from (Zhu et al., 2018) and run it on the validation videos. Table 2 (c) shows the performance of various methods, evaluated at different clip lengths and frame rates. For this task we also experiment with TSN (Wang et al., 2016b), though it ends up performing significantly worse than R3D. Note that this contrasts with standard video datasets (Kay et al., 2017), where it tends to perform similar to R3D (Xiong, 2017). We also experiment with the flow modality and observe it obtains even lower performance, which is expected as this task requires recognizing objects which is much harder from flow. Again, note that flow models obtain similar if not better performance as RGB on standard datasets (Kay et al., 2017; Xiong, 2017). We also note higher performance on considering longer clips with higher sample rate. This is not surprising as a task like this would require long term temporal reasoning, which is aided by looking at longer videos. This is also reinforced by the observation that using LSTM for aggregation leads to a major improvement in performance for most models. Finally, the tracking approach also only solves about a third of the videos, as even the state of the art tracker ends up drifting due to occlusions and contain operations.
In Table 4, we ablate the performance with respect to the underlying grid granularity, with 6 × 6 being the default used in Table 2 (c). We observe tracking is a stronger baseline as the localization task gets more fine-grained. Finally in Table 3 we compare performance of some of these models on existing benchmarks and CATER.
Analysis: Having close control over the dataset generation process enables us to perform diagnostics impossible with any previous dataset. We use the R3D+NL, 32-frame, static camera model with average (or LSTM, when specified) pooling for all following visualizations. We first analyze aggregate performance of our model over multiple bins in Figure 4, and observe some interesting phenomena. (a) Performance drops if the snitch keeps moving until the end. This makes sense: if the snitch reaches its final position early in the video, models have a lot more frames to reinforce their hypothesis of its final location. Between LSTM and avg-pooling, LSTM is much better able to handle the motion of the snitch, as expected. Perhaps not surprisingly, the tracker is much less effected by snitch movement, indicating the power of such classic computational pipelines for longterm spatiotemporal understanding. (b) Drops if the snitch is contained in the end. Being contained in the final frame makes the snitch harder to spot and track (just like the cups and ball game!), hence the lower performance.
Next, we visualize the videos that our models gets right or wrong. We sort all validation videos based on the softmax confidence score for the ground truth class, and visualize the top and bottom six in Figure 5 (full video in supplementary). We find that the easiest videos for avg-pooled model tend to be ones with little snitch motion, i.e. the object stays at the position it starts off in. On the other hand, the LSTM-aggregated model fares better with snitch motion, as long as it happens early in the video. The hardest videos for both tend to be ones with sudden motion of the snitch towards the end of the video, as shown by the bright golden trail denoting the motion towards the end (better viewed in supplementary video). These observations are supported by the quantitative plots in Figure 4 (a) and (c).

heading: 5 CONCLUSION

text:
We use CATER to analyze several leading network designs on hard spatiotemporal tasks. We find most models struggle on our proposed dataset, especially on the snitch localization task which requires long term reasoning. Interestingly, average pooling clip predictions or short temporal cues (optical flow) perform rather poorly on CATER, unlike most previous benchmarks. Such temporal reasoning challenges are common in the real world (eg. Fig. 1 (a)), and solving those would be the cornerstone of the next improvements in machine video understanding. We believe CATER would
serve as an intermediary in building systems that will reason over space and time to understand actions. That said, CATER is, by no means, a complete solution to the video understanding problem. Like any other synthetic or simulated dataset, it should be considered in addition to real world benchmarks. While we have focused on classification tasks for simplicity, our fully-annotated dataset can be used for much richer parsing tasks such as spacetime action localization. One of our findings is that while high-level semantic tasks such as activity recognition may be addressable with current architectures given a richly labeled dataset, “mid-level” tasks such as tracking still pose tremendous challenges, particularly under long-term occlusions and containment. We believe addressing such challenges will enable broader temporal reasoning tasks that capture intentions, goals, and causal behavior.

heading: ACKNOWLEDGMENTS

text: Authors would like to thank Ishan Misra for many helpful discussions and help with systems. This research is based upon work supported in part by NSF Grant 1618903.

heading: B TRAIN/VAL DISTRIBUTIONS

text:
Figure 6 shows the data distribution over classes for each of the tasks.
C VIDEO VISUALIZATION
The supplementary video3 visualizes:
1. Sample videos from the dataset (with and without camera motion). 2. Easiest and hardest videos for task 3. We rank all validation videos for task 3 based on
their softmax probability for the correct class. We show the top-6 (easiest) and bottom-6 (hardest) for 32-frame stride-8 non-local + LSTM model. We observe the hardest ones involve sudden motion towards the end of the video. This reinforces the observation made in Figure 5(a) in the main paper, that videos where snitch keeps moving till the end are the hardest. If the snitch stops moving earlier, models have more evidence for the final location of the snitch, making the task easier.
3. Tracking results. We visualize the results of tracking the snitch over the video as one approach to solving task 3. We observe that while it works in the simple scenarios, it fails when there is a lot of occlusion or complex contain operations.
4. Model bottom-up attention. We visualize where does the model look for Task 3. As suggested in (Malinowski et al., 2018), we visualize the l2-norm of the last layer features from our 32-frame stride-8 non-local model on the center video crop. The deep red color denotes large norm value at that spatiotemporal location. We find that the model automatically learns to focus on the snitch towards the end of clips, which makes sense as that is the most important object for solving the localization task.
3https://rohitgirdhar.github.io/CATER/assets/suppl/video.mp4

title: CATER: A DIAGNOSTIC DATASET FOR COMPOSITIONAL ACTIONS & TEMPORAL REASONING


summaries: In this work authors consider a problem of 'model compatibility' of GANs, i.e. usefullness of the generated samples for classification tasks. Proposed 'Boundary Calibration' GAN attempts to tackle this issue by adding non-adversarial terms to discriminator, obtained as outputs of the classifiers trained on the original data. For evaluation, it is proposed to compare accuracies obtained by classifiers trained on generated and on real data (termed 'relative acurracy'). Experiments show that the proposed methods improve such scores.

summaries: In this paper the authors propose a method for improving "model compatibility" in GANs. For this reason they add to the loss of the generation procedure a term that depends on the maximum mean discrepancy between the following datasets: (1) the output of a classifier with input the real dataset, (2) the output of the same classifier with input GAN-generated samples. They authors show that in essentially all the datasets they tried, the model compatibility of the produces generator is increased after adding the aforementioned cost, while the visual quality of the data is not decreased.

summaries: This paper aims at training a GAN that can generate data matches the real data distribution well especially at the boundaries of the classifiers. A Boundary-Calibration loss (BC-loss) base on multi pretrained classifiers is introduced to match the statistics between the distributions of original data and generated data. The motivation is interesting. The story is clearly explained. However, the experiments part is weak.

abstractText: Generative Adversarial Networks (GANs) is a powerful family of models that learn an underlying distribution to generate synthetic data. Many existing studies of GANs focus on improving the realness of the generated image data for visual applications, and few of them concern about improving the quality of the generated data for training other classifiers—a task known as the model compatibility problem. Literature also show that some GANs often prefer generating ‘easier’ synthetic data that are far from the boundaries of the classifiers, and refrain from generating near-boundary data, which are known to play an important roles in training the classifiers. To improve GAN in terms of model compatibility, we propose Boundary-Calibration GANs (BCGANs), which leverage the boundary information from a set of pre-trained classifiers using the original data. In particular, we introduce an auxiliary Boundary-Calibration loss (BC-loss) into the generator of GAN to match the statistics between the posterior distributions of original data and generated data with respect to the boundaries of the pre-trained classifiers. The BC-loss is provably unbiased and can be easily coupled with different GAN variants to improve their model compatibility. Experimental results demonstrate that BCGANs not only generate realistic images like original GANs but also achieves superior model compatibility than the original GANs.

authors:
Rick JH Chang
Chun-Liang Li
Barnabas Poczos
BVK Vijaya Kumar
Aswin C Sankaranarayanan

title: One network to solve them all–solving linear inverse problems using deep projection models

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Wei-Cheng Chang
Chun-Liang Li
Yiming Yang
Barnabás Póczos

title: Kernel change-point detection with auxiliary deep generative models

venue: arXiv preprint arXiv:1901.06077,

authors:
Ian J. Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron C. Courville
Yoshua Bengio

title: Generative adversarial nets

venue: In Advances in Neural Information Processing Systems,

authors:
Arthur Gretton
Karsten M. Borgwardt
Malte J. Rasch
Bernhard Schölkopf
Alexander J. Smola

title: A kernel two-sample test

venue: J. Mach. Learn. Res.,

authors:
Ishaan Gulrajani
Faruk Ahmed
Martı́n Arjovsky
Vincent Dumoulin
Aaron C. Courville

title: Improved training of wasserstein gans

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Identity mappings in deep residual networks

venue: In Computer Vision - ECCV 2016 - 14th European Conference,

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: Gans trained by a two time-scale update rule converge to a local nash equilibrium

authors:
R Devon Hjelm
Athul Paul Jacob
Tong Che
Adam Trischler
Kyunghyun Cho
Yoshua Bengio

title: Boundary-seeking generative adversarial networks

venue: arXiv preprint arXiv:1702.08431,

authors:
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou
Alexei A. Efros

title: Image-to-image translation with conditional adversarial networks

venue: IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Tero Karras
Timo Aila
Samuli Laine
Jaakko Lehtinen

title: Progressive growing of gans for improved quality, stability, and variation

venue: In 6th International Conference on Learning Representations,

authors:
Taeksoo Kim
Moonsu Cha
Hyunsoo Kim
Jung Kwon Lee
Jiwon Kim

title: Learning to discover cross-domain relations with generative adversarial networks

venue: In Proceedings of the 34th International Conference on Machine Learning,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Christian Ledig
Lucas Theis
Ferenc Huszár
Jose Caballero
Andrew Cunningham
Alejandro Acosta
Andrew Aitken
Alykhan Tejani
Johannes Totz
Zehan Wang

title: Photo-realistic single image super-resolution using a generative adversarial network

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Chun-Liang Li
Wei-Cheng Chang
Yu Cheng
Yiming Yang
Barnabás Póczos

title: MMD GAN: towards deeper understanding of moment matching network

authors:
Mario Lucic
Karol Kurach
Marcin Michalski
Sylvain Gelly
Olivier Bousquet

title: Are gans created equal? a large-scale study

venue: In Advances in neural information processing systems,

authors:
Yonghong Luo
Xiangrui Cai
Ying Zhang
Jun Xu

title: Multivariate time series imputation with generative adversarial networks

authors:
Mehdi Mirza
Simon Osindero

title: Conditional generative adversarial nets

venue: arXiv preprint arXiv:1411.1784,

authors:
Sebastian Nowozin
Botond Cseke
Ryota Tomioka

title: f-gan: Training generative neural samplers using variational divergence minimization

authors:
Augustus Odena
Christopher Olah
Jonathon Shlens

title: Conditional image synthesis with auxiliary classifier gans

authors:
Noseong Park
Mahmoud Mohammadi
Kshitij Gorde
Sushil Jajodia
Hongkyu Park
Youngmin Kim

title: Data synthesis based on generative adversarial networks

authors:
Deepak Pathak
Philipp Krahenbuhl
Jeff Donahue
Trevor Darrell
Alexei A Efros

title: Context encoders: Feature learning by inpainting

authors:
Ofir Press
Amir Bar
Ben Bogin
Jonathan Berant
Lior Wolf

title: Language generation with recurrent generative adversarial networks without pre-training

venue: arXiv preprint arXiv:1706.01399,

authors:
Alec Radford
Luke Metz
Soumith Chintala

title: Unsupervised representation learning with deep convolutional generative adversarial networks

venue: In 4th International Conference on Learning Representations,

authors:
Tim Salimans
Ian J. Goodfellow
Wojciech Zaremba
Vicki Cheung
Alec Radford
Xi Chen

title: Improved techniques for training gans

authors:
Rui Shu
Hung Bui
Stefano Ermon

title: Ac-gan learns a biased distribution

venue: In NIPS Workshop on Bayesian Deep Learning,

authors:
Jiajun Wu
Chengkai Zhang
Tianfan Xue
Bill Freeman
Josh Tenenbaum

title: Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling

authors:
Lei Xu
Maria Skoularidou
Alfredo Cuesta-Infante
Kalyan Veeramachaneni

title: Modeling tabular data using conditional gan

venue: arXiv preprint arXiv:1907.00503,

authors:
Lantao Yu
Weinan Zhang
Jun Wang
Yong Yu

title: Seqgan: Sequence generative adversarial nets with policy gradient

venue: In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence,

authors:
Jun-Yan Zhu
Taesung Park
Phillip Isola
Alexei A. Efros

title: Unpaired image-to-image translation using cycle-consistent adversarial networks

venue: In IEEE International Conference on Computer Vision,

heading: 1 INTRODUCTION

text:
The success of machine learning relies on not only the advances of different models (e.g. deep learning) but also data with sufficient quality and quantity. Nowadays, companies spend tremendous efforts and expense collecting data to build their products. To better solve complicated real-world problems with public or third-party machine learning experts, many companies now needs release some data sets for competitions (e.g. Kaggle) or proof-of-concept purposes. However, considering the costs of collecting data, companies may not be willing to release the dataset if possible. As a result, a technique which can generate synthetic data with properties similar to the original data is in demand. To be specific, we are looking for generating a dataset with the property that machine learning models trained on the generated dataset can exhibit similar performance to ones trained on the original data. This property is called model compatibility (Park et al., 2018) or machine learning efficacy (Xu et al., 2019). The organizations can share the generated data with high model compatibility to the public and enjoy the solution derived from it without leaking the real dataset.
When it comes to data generation, generative adversarial networks (GANs, Goodfellow et al. 2014) is the most popular family of generative algorithms because of its impressive performance on generating realistic images (Karras et al., 2018). In GANs, the generator is trained via minimizing a neural network (discriminator) defined probability divergence (Goodfellow et al., 2014; Arjovsky et al., 2017; Nowozin et al., 2016). In addition to image generation, GANs are also widely used in other applications, such as style transfer (Isola et al., 2017; Zhu et al., 2017; Kim et al., 2017) and image processing (Pathak et al., 2016; Ledig et al., 2017; Chang et al., 2017), and generating different types of data, including time series (Luo et al., 2018; Chang et al., 2019), text (Yu et al., 2017; Press et al., 2017), point clouds (Li et al., 2018), voxels (Wu et al., 2016) and tabular data (Park et al., 2018; Xu et al., 2019).
Although GANs are versatile as mentioned above, most of their development focus on the metrics such as quality and diversity of the data Salimans et al. (2016); Heusel et al. (2017); Lucic et al. (2018). Generating high model compatibility data via GANs is still under explored. The pioneering work (Xu et al., 2019) first shows that data generated from conditional GANs (Mirza & Osindero, 2014) enjoys better model compatibility than VAEs (Kingma & Welling, 2013). So we wonder can we improve the model compatibility if we consider information from the models trained on the original data? For example, Wasserstein GAN (WGAN, Arjovsky et al. 2017) performs a meanmatching between the distribution of real data and generated data. However, only mean-matching is sometimes not enough to learn the whole distribution especially for those boundary cases. Apparently, if a GAN knows the boundary between different classes, it may be able to generate instances which are close to the boundary with correct labels. These boundary points will guide a classifier to learn the correct decision boundary.
In this work, we try to improve GANs with regards to model compatibility in classification problems. We use a set of pre-trained classifiers to obtain multiple decision boundaries. Then use an auxiliary loss function called Boundary-Calibration loss (BC-loss) to calibrate the generating distribution according to the decision boundaries of these pre-trained classifiers. The main contributions of this work are:
• In Section 2, we propose a way to evaluate model compatibility in classification problems. We consider a variety of machine learning algorithms and average the performance to obtain a comprehensive metric.
• In Section 4, we propose a loss function called Boundary-Calibration loss (BC-loss) which helps typical GANs to learn a distribution with better model compatibility. The loss considers the decision boundaries of pre-trained classifiers and minimizes the maximum mean discrepancy (MMD, Gretton et al. 2012) between the original dataset and the generated dataset. In addition, we show that optimizing the BC-loss would not change the optimal solution of the original GAN, but it reduces the feasible set to ensure the model compatibility.
• In Section 5, we demonstrate how BC-loss affects the boundary of the generated data with a two-dimensional toy dataset. We also show that the BC-loss improves model compatibility of the generated data with different types of classifiers and a variety of datasets. Finally, we inspect the feature selection results to examine how the interpretation of machine learning models may be effected.
Last, in Section 3, we discuss some works which are similar to our work and describe how does our work differ from them.

heading: 2 MODEL COMPATIBILITY IN CLASSIFICATION

text:
In this work, we focus on generating data for fully-supervised classification learning. Given a dataset D = {(xi, yi)}ni=1, where xi ∈ X represents features of an instance, yi = f(xi) ∈ Y represents the corresponding label of xi according f : X → Y , and (xi, yi) ∼ PD, a learning algorithm A : (X ,Y)n → H learns a hypothesis h ∈ H to approximate the mapping function, i.e. A(D) = h ≈ f . Our goal is to obtain a generator G which generates a synthetic dataset D′ = {(x′j , y′j)}mj=1 such that A(D′) = h′ ≈ h. We call this property model compatibility as proposed in Park et al. (2018). To measure the model compatibility of a generated dataset quantitatively, we consider the performance of a classifier trained on the generated dataset comparing to the one trained on the real dataset. We evaluate the accuracy on a separate test dataset to indicate the performance of a given classifier. In addition, we calculate relative accuracy by scaling the test accuracy of the classifier trained on the generated dataset by the accuracy of the classifier trained on the real dataset. The relative accuracy allows us to average the results from different machine learning algorithms more fairly. The final evaluation is :
1 |A| ∑ A∈A acc(h′, D(t)) acc(h,D(t)) , (1)
where A is a set of learning algorithms ,D(t) = {(x(t)i , y (t) i )}Ni=1 is the test dataset, and acc(h,D(t)) is the accuracy of hypothesis h on test data D(t). We can determine A as a wide variety of learning algorithms to make the metric provide a more comprehensive measurement of model compatibility.

heading: 3 RELATED WORK

text:
Research about generating data for classification can be divided into two categories: formulation and architecture. For formulation, Conditional GAN (CGAN, Mirza & Osindero 2014 ) is an intuitive way to generate instances with corresponding labels. We can learn the distribution of labels by counting and sample the instances from CGAN conditionally. Auxiliary Classifier GAN (ACGAN, Odena et al. 2017) is considered as a better way for conditional generation. It uses an auxiliary classifier to provide information about the boundary between each classes. However, ACGAN has been proved that the objective is biased so it tends to generate data with lower entropy for the auxiliary classifier (Shu et al., 2017). Thus, the lose of instances near the decision boundary may worsen the model compatibility. In this work, we use CGAN along with the proposed BC-loss to generate data with model compatibility.
On the other hand, the other line of research focuses on generating data with different network architecture or data processing procedure. Recent works that also consider model compatibility are Table GAN (Park et al., 2018) and Tabular GAN (Xu et al., 2019). Table GAN focuses on the privacy of generated data and thus their is a trade off between privacy and model compatibility. To achieve privacy preserving, they do not improve the model compatibility compared to the original GAN. On the other hand, Tabular GAN puts emphasis on increasing model compatibility of generated data. They propose a framework with a more complicated data processing procedure and use LSTM to better parameterize the target distribution. In contrast to these works, our work focus on the formulation of GANs and can be applied to most variants of GANs, including Table-GAN and Tabular GAN. Moreover, while these former works only focus on tabular data, our BC-loss is applicable to generate image datasets as well.
Some GAN variants are named similarly to our work but they pay attention to different problems. For example, the boundary described in boundary-seeking GAN (Hjelm et al., 2017) means the decision boundary of the discriminator rather than the decision boundary for the supervised labels. To the best of our knowledge, we are the first work trying to improve model compatibility by modifying the formulation of GANs.

heading: 4 BOUNDARY-CALIBRATION GAN

text: To achieve better model compatibility of GAN, we propose an auxiliary GAN loss which we call boundary-calibration loss (or BC-loss). We assume that we have a set of pre-trained classifiers which are well-trained on the original dataset. The BC-loss helps GANs to calibrate the distribution with respect to the distribution of decision values predicted by pre-trained classifiers. The calibration leads to more accurate data generation near the decision boundary and thus enabling a machine learning algorithm to learn a similar hypothesis to one that learns from the original dataset. To ease the complexity of learning to generate (x, y) jointly, we infer P (y) by counting the proportion of each class in the original dataset and train a conditional generator G such that G(z, y) ∼ PX|y , where PX|y is the conditional data distribution and z ∼ PZ is the initial randomness such as Gaussian distribution. Therefore we can generate (x, y) by sampling y ∼ P (y) and G(z, y).

heading: 4.1 BOUNDARY CALIBRATION

text:
Given a pre-trained classifier C, we hope the generated dataset will adopt the same statistics as the original dataset while considering the decision boundary of C. To include the information about the boundary, we calculate posterior PC(y | xi) from the decision values predicted by the classifier. In practice, we can apply a softmax function to the outputs of a classifier to obtain the posterior. The posterior provides information of an instance from the classifier’s aspect. Therefore, given the real dataset X = {x1,x2, ...,xn}, we can obtain a set of posterior vector C(X) = (PC(y | x1), PC(y | x2), ..., PC(y | xn)). To generate data X ′ with the same distribution of posteriors to the boundary,
we match the statistics of C(X) and C(X ′) by optimizing a distance M :
LBC(X,X ′, C) =M(C(X), C(X ′)) (2)
Here M can be any distance metric which measures the distance between two sets of samples. In statistics, distinguishing whether two sets of samples are from the same distribution is called TwoSample Test. A classical solution to two-sample test is kernel maximum mean discrepancy (MMD, Gretton et al. 2012). The idea is to compare the statistics between the two sets of samples. If the statistics are similar then these two sets might be sampled from the same distribution. Given two sets of samples X = {xi}ni=1 and Y = {yj}nj=1, an unbiased estimator of MMD with kernel k is defined as:
M̂k(X,Y ) = 1( n 2 ) ∑ i 6=i′ k(xi,xi′)− 2( n 2 ) ∑ i 6=j k(xi,yj) + 1( n 2 ) ∑ j 6=j′ k(yj ,yj′) (3)
In practice, we use Gaussian kernel k(x,x′) = exp(‖x − x′‖2) in MMD since Gaussian kernel is a characteristic kernel which ensures that the distance is zero if and only if the two distributions are the same (Gretton et al., 2012).
The BC-loss can be applied in generator of any GAN variants to improve the model compatibility. In addition, to better fit the real unknown boundary, we can use multiple classifiers to calibrate the distributions from different aspects. As a result, for a loss function of generator LG, we can modify the loss to be:
L̂G = LG + λ |C| ∑ C∈C M̂k(C(X), C(G(Z,Y))) (4)
where Z is a set of noises, C is a set of pre-trained classifiers and λ is a hyper-parameter to control the weight of BC-loss.

heading: 4.2 ANALYSIS OF OPTIMAL SOLUTION

text:
Next we prove that adding our proposed BC-loss would not change the optimal solution of the original objective. Here we assume the loss of the generator LG achieves its optimal value in the its GAN objectives LG if and only if the distribution of G(z, y) recovers PX|y for all y ∈ Y , which holds for the vanilla GAN (Goodfellow et al., 2014) and most of other GAN variants. Theorem 1 (Gretton et al. 2012). Given a kernel k, if k is a characteristic kernel, thenMk(P,Q) = 0 ⇐⇒ P = Q. Theorem 2 (Equivalence of optimal solution). G is an optimal solution ofLG ⇐⇒ G is an optimal solution of L̂G
Proof. (⇒) According to the assumption, G is an optimal solution of LG implies G(Z, y) recovers PX|y for all y ∈ Y . Therefore, PC(X) = PC(G(Z,Y)) and Mk(PC(X), PC(G(Z,Y))) = 0 by Theorem 1. Now L̂G = LG + 0 = LG and G is an optimal solution of LG, so G is also an optimal solution of L̂G. (⇐) Since LBC ≥ 0, we have L̂G = LG + LBC ≥ LG. From above, we know L̂G(G) = 0 if G = PX . Thus, for an optimal solution G∗, 0 ≥ L̂G(G∗) ≥ LG(G∗) ≥ 0, which implies L̂G(G∗) = LG(G∗) = 0. Therefore, G∗ is also an optimal solution of LG.
The proof shows that the proposed BC-loss does not change the optimal solution of the original optimization problem. However, we can consider BC-loss as a Lagrangian constraint which restricts the solution to a subspace where the generator owns higher model compatibility .

heading: 4.3 COMPARISON TO MMD GAN

text:
MMD GAN (Li et al., 2017) is a variant of GAN where the generator tries to minimize the MMD between generated data and original data and the discriminator learns a kernel which maximizes the MMD. Though the formulation of MMD GAN and BC-loss are similar, they still do not conflict because MMD GAN do not known the information about the classifier and the objective of
MMD GAN would not lead the discriminator to a classifier. Therefore, BC-loss may still improve MMD GAN by guiding the generator to not generate points across the boundary. To understand the improvement in MMD GAN from BC-loss, we use MMD GAN as one of the baselines in our experiments.

heading: 5 EXPERIMENTS

text: In this section, we use a toy dataset to illustrate how the proposed method improves the model compatibility. To be more realistic, we provide more comprehensive results for four different realworld dataset from UCI dataset repository (Dua & Graff, 2017): Adult, Connect-4, Covertype and Sensorless. We then show our method is also applicable in image dataset: MNIST and Cifar10 without losing the image quality. In addition, we investigate the results of feature selections on the generated dataset to see whether the generated data can preserve the interpretation of machine learning models.

heading: 5.1 EXPERIMENTAL SETTINGS

text:
EVALUATION
In this work, we focus on model compatibility of generated datasets. We use a wide variety of machine learning algorithms including linear SVM, decision tree (DT), random forest (RF), and multi-layer perception (MLP) to evaluate the model compatibility. As described in Section 2, we evaluate the relative accuracy for each type of machine learning model, where the relative accuracy is calculated by dividing the accuracy of classifier trained on generated data to the accuracy of classifier trained on original data.
COMPARED METHODS
We take Wasserstein GAN (WGAN) and MMD GAN as our baselines to evaluate the effectiveness of the proposed boundary-calibration technique. We denotes their counterparts with BC-loss as BWGAN and BMMDGAN respectively. All of the methods use gradient penalty to enforce the Lipschitz constraint on the discriminator (Gulrajani et al., 2017; Li et al., 2017). To achieve conditional data generation as described in Section 4, we add an embedding layer to learn the embedding vector for each class. The embedding vector is concatenated as additional input features for both generators and discriminators on UCI datasets. For image datasets, the embedding vectors are used as described in Mirza & Osindero (2014).

heading: 5.2 2D TOY DATASET

text:
We use a 2D toy dataset with two classes to illustrate the results generated by different GAN methods in Figure 1. Figure 2a shows the distribution of the original training data. We use these generated data to train a random forest and depict the decision boundary by different background color. From Figure 2b, we can see that although ACGAN can make use of the auxiliary classifier during training,
it learns a biased distribution that push the generated data away from the boundary. The large margin between the two clusters brings more uncertainty to the decision boundary and thus leads to worse test accuracy. In Figure 2c, WGAN approximates the original distribution well in the center part of the two cluster, but do not get a clear boundary between the two classes. It generates some ambiguous points near the boundary that would confuse the classifier. Finally, our BWGAN generates points near the boundary more precisely, as shown in Figure 2c.

heading: 5.3 UCI DATASET

text:
We evaluate our proposed BC-GAN on four datasets from UCI repository. The attributes of the datasets can be found in Appendix A. Discrete features are processed to one-hot encoding and continuous features are scaled to [0, 1]. For each dataset, we train six multi-layer perceptrons with a random split of half of training data as pre-trained classifiers. In these experiments, generators and discriminators are consist of 3 fully-connected hidden layer with 128 units. A logistic function is applied to the output layer of generators to generate features within 0 and 1. The weight of BC-loss is set to be λ = 100 for all datasets.
Table 1 summarize the comparison between different methods. We calculate the relative accuracy of different machine learning models mentioned in Section 5.1 and average the relative accuracy to indicate the model compatibility of generated data for each dataset. The table shows that the proposed BC-loss improves the accuracy of classifiers generally compared to original WGAN and MMD GAN. Moreover, ACGAN performs worst on three out of four datasets and exhibit a significant deficiency though it is proved to have the state-of-the-art generation quality. This again prove that the biased objective of ACGAN worsen the model compatibility seriously.The breakdown results and real accuracy are provided in Appendix B.
To further investigate the advantage of boundary-calibration, we visualize the generated results of Sensorless in Figure 2. We train a fully-connected neural network with a 2-units hidden layer before the output layer to project the generated samples to a 2-dimensional embedding space. The projection classifier is well-trained and achieves over 99% testing accuracy so we can use it to determine whether a sample is generated with incorrect label. The figure shows that there are less mislabeled data generated by BWGAN, especially at the center and the bottom-left region. The fact indicates that boundary-calibration helps GANs generate labeled data more accurately, which may lead to the improvement of classification accuracy.

heading: 5.4 INTERPRETABILITY

text:
In addition to accuracy, it is also important that the model trained on generated data should give us the same interpretation of a model trained on the original data. We investigate the interpretability by two common feature selection techniques. First, we train two random forests on the generated and original dataset respectively. Each random forest can provide the importances of the features. We evaluate the consistency of interpretation by calculating precision at Kth, which means how many features ranked top-k in random forest trained on original data are in the top-k importance feature of the random forest trained on generated data. The results are shown in Table 2. We provide the results of training a classifier on the same original data with a different random seed as REAL for comparison. The effect of BC-loss is not significant in this aspect. However, the scores of ACGAN drop seriously, which means training a classifier on data generated by ACGAN is somehow dangerous because the meaning of model may be totally different.
Another way to select feature is training a linear model with `1 regularization. In Table 3 we use linear SVM with `1 regularization to select features. Then we calculate the F1 score of features selected by classifiers trained on generated data to known how similar between the two sets of features selected by classifiers trained on original and generated dataset. The results again shows that using boundary-calibration does not has significant effect to feature selection and ACGAN is not proper to generated data for training.

heading: 5.5 IMAGE DATASET

text:
We further use MNIST and CIFAR-10 dataset to investigate the effectiveness of boundarycalibration on image datasets. For MNIST, we train six 4-layer convolution neural networks (CNN) with random sampling half of training data as pre-trained classifiers, and use the same classifier set in Section 5.1 to evaluate model compatibility. For CIFAR-10, we use ResNet56v2 (He et al., 2016) to obtain three pre-trained classifier and evaluate on CNN and ResNet56v2. In both task, we use DCGAN (Radford et al., 2016) as network structure in all GANs. The weight of BC-loss is set to be λ = 1 for these two datasets.
Table 4 and Table 5 show the relative accuracy of classifiers trained on generated data. The proposed BWGAN still outperforms WGAN with better accuracy in general. The results generated by WGAN and BWGAN are pictured in Figure 3. The Inception score and Frechet Inception Distance (FID) for CIFAR-10 are also provided in the caption of Figure 3. Though the difference of quality between the images generated from WGAN and BWGAN is not significant in visual, the quantitative scores for quality of generated samples of CIFAR-10 are slightly improved. The results indicate that even though our method seems not improve the image quality, it is still able to improve the model compatibility without losing image quality.
Table 4: Breakdown results on MNIST dataset.
REAL WGAN BWGAN
DT (d=10) 86.6 54.5 (63.0) 48.3 (55.8)) DT (d=20) 88.0 34.5 (39.2) 48.2 (54.8) Linear SVM 88.0 34.5 (39.2) 48.2 (54.8) MLP (100) 97.6 96.2 (98.5) 96.8 (99.2) MLP (200x2) 97.9 96.7 (98.8) 96.1 (98.2) RF (n=10, d=10) 92.5 75.5 (81.7) 83.7 (90.6) RF (n=10, d=20) 94.7 67.5 (71.3) 71.5 (75.4)
Avg. 100.0 70.2 75.5
Table 5: Breakdown results on CIFAR10 dataset.
REAL WGAN BWGAN
CNN 70.8 63.5 (89.8) 63.0 (89.1) Resnet56v2 77.5 48.8 (62.9) 51.3 (66.2)
Avg. 100.0 76.4 77.6

heading: 6 DISCUSSION

text: We introduce an auxiliary loss in GANs which improves the model compatibility of generated dataset. We prove the new loss is unbiased and is applicable to all variants of GAN to improve model compatibility. We further demonstrate that our method has clear advantages with a variety of machine learning models trained on generated dataset. In addition, we investigate the results of feature selection and found that the BC-loss doesn’t effect the interpretation of machine learning models. While this work only focus on classification problem, generating data for regression problem is also worth studying. We hope our work will open the path for GANs with better model compatibility so that synthetic data can be more useful in practice.

heading: B DETAIL RESULT

heading: A DATASET INFORMATION

text:
REAL ACGAN WGAN BWGAN MMDGAN BMMDGAN
DT (d=10) 96.3 (100.0) 75.8 (78.7) 77.9 (80.8) 87.2 (90.5) 82.3 (85.5) 82.6 (85.8) DT (d=20) 98.4 (100.0) 75.4 (76.7) 68.3 (69.5) 89.6 (91.1) 81.3 (82.6) 83.8 (85.2) Linear SVM 98.4 (100.0) 75.4 (76.7) 68.3 (69.5) 89.6 (91.1) 81.3 (82.6) 83.8 (85.2) MLP (100) 93.6 (100.0) 72.8 (77.8) 87.8 (93.8) 89.8 (96.0) 82.4 (88.0) 84.9 (90.6) MLP (200x2) 98.7 (100.0) 76.8 (77.8) 90.5 (91.7) 93.7 (94.9) 85.4 (86.5) 87.7 (88.8) RF (n=10, d=10) 98.4 (100.0) 76.2 (77.5) 92.1 (93.6) 92.5 (94.1) 87.9 (89.4) 88.7 (90.2) RF (n=10, d=20) 99.8 (100.0) 77.1 (77.3) 93.3 (93.6) 95.6 (95.8) 89.2 (89.4) 90.8 (91.0)
Avg. 100.0 77.5 84.6 93.3 86.3 88.1