summaries: This paper investigates the problem of extracting a meaningful state representation to help with exploration in RL, when confronted to a sparse reward task. The core idea consists in identifying controllable (learned) features of the state, which in an Atari game for instance typically corresponds to the position of the player-controlled character / vehicle on the screen. Once this position is known (as x, y coordinates on a custom low-resolution grid), one can use existing count-based exploration mechanisms to encourage the agent to visit new positions (NB: in addition to the x, y coordinates, extra information is also used to disambiguate the states for counting purpose, namely the current score and the state’s cluster index obtained with a basic clustering scheme). To find the position, the algorithm trains one inverse dynamics model per x, y cell on the grid: each model tries to predict the action taken by the agent given two consecutive states, both represented by their feature map (at coordinate x, y) learned by a convolutional network applied to the pixel representation. The outputs of these inverse dynamics models are combined through an attention mechanism to output the final prediction for the action: the intuition is that the attention model will learn to focus on the grid cell with best predictive power (for a given state), which should correspond to where the controllable parts of the state are. Experiments on several Atari games (including Montezuma’s Revenge) indeed show that this mechanism is able to track the true agent’s coordinates (obtained from the RAM state) reasonably well. Using these coordinates for count-based exploration (in A2C) also yields significantly better results compared to vanilla A2C, and beats several previously proposed related techniques for exploration in sparse reward settings.

summaries: This paper introduces contingency-aware exploration by employing attentive dynamics model (ADM). ADM is learned in self supervised manner in an online fashion and only using pure observations as the agents policy is updated. This approach has clear advantages to earlier proposed count based techniques where agent's curiosity is incentivized for exploration. Proposed technique provides an important insight into how to approach such challenging tasks where the rewards are very sparse. Not only it achieves state of the art results with convincing empirical evidence but also authors make a good job of providing details of their specific modelling techniques for training challenges. They make a good job of comparing and contrasting the contingency-awareness by ADM to earlier proposed methods such as intrinsic motivation and self-supervised dynamics model. Overall exposition is clear with well explained results. The proposed idea raises interesting questions for future work.

abstractText: This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning. To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games. The ADM is trained in a self-supervised fashion to predict the actions taken by the agent. The learned contingency information is used as a part of the state representation for exploration purposes. We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards.1 For example, we report a state-of-the-art score of >11,000 points on MONTEZUMA’S REVENGE without using expert demonstrations, explicit high-level information (e.g., RAM states), or supervisory data. Our experiments confirm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations.

authors:
Joshua Achiam
Shankar Sastry

title: Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

venue: arXiv preprint arXiv:1703.01732,

authors:
Pulkit Agrawal
Ashvin Nair
Pieter Abbeel
Jitendra Malik
Sergey Levine

title: Learning to Poke by Poking: Experiential Learning of Intuitive Physics

authors:
Frank Baeyens
Paul Eelen
Omer van den Bergh

title: Contingency awareness in evaluative conditioning: A case for unaware affective-evaluative learning

venue: Cognition and emotion,

authors:
Dzmitry Bahdanau
Kyunghyun Cho
Yoshua Bengio

title: Neural Machine Translation by Jointly Learning to Align and Translate

venue: In ICLR,

title: Intrinsic motivation and reinforcement learning

authors:
Marc G Bellemare
Joel Veness
Michael Bowling

title: Investigating Contingency Awareness Using Atari 2600 Games

venue: In AAAI,

authors:
Marc G. Bellemare
Yavar Naddaf
Joel Veness
Michael Bowling

title: The Arcade Learning Environment: An Evaluation Platform for General Agents

venue: Journal of Artificial Intelligence Research

authors:
Marc G. Bellemare
Sriram Srinivasan
Georg Ostrovski
Tom Schaul
David Saxton
Remi Munos

title: Unifying Count-Based Exploration and Intrinsic Motivation

authors:
Emmanuel Bengio
Valentin Thomas
Joelle Pineau
Doina Precup
Yoshua Bengio

title: Independently Controllable Features

venue: arXiv preprint arXiv:1703.07718,

authors:
Yuri Burda
Harri Edwards
Deepak Pathak
Amos Storkey
Trevor Darrell
Alexei A. Efros

title: Large-scale study of curiosity-driven learning

venue: arXiv preprint arXiv:1808.04355,

authors:
Yuri Burda
Harrison Edwards
Amos Storkey
Oleg Klimov

title: Exploration by random network distillation

title: Similarity estimation techniques from rounding algorithms

venue: In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing,

authors:
Nat Dilokthanakul
Christos Kaplanis
Nick Pawlowski
Murray Shanahan

title: Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

venue: arXiv preprint arXiv:1705.06769,

authors:
Hugh Durrant-Whyte
Tim Bailey

title: Simultaneous Localization and Mapping: Part I

venue: IEEE robotics & automation magazine,

authors:
Benjamin Eysenbach
Abhishek Gupta
Julian Ibarz
Sergey Levine

title: Diversity is All You Need: Learning Skills without a Reward Function

authors:
Rein Houthooft
Xi Chen
Yan Duan
John Schulman
Filip De Turck
Pieter Abbeel

title: VIME: Variational Information Maximizing Exploration

authors:
Max Jaderberg
Volodymyr Mnih
Wojciech Marian Czarnecki
Tom Schaul
Joel Z Leibo
David Silver
Koray Kavukcuoglu

title: Reinforcement Learning with Unsupervised Auxiliary Tasks

authors:
Brian Kulis
Michael I . Jordan

title: Revisiting k-means: New Algorithms via Bayesian Nonparametrics

venue: In ICML,

authors:
Marlos C Machado
Marc G Bellemare
Erik Talvitie
Joel Veness
Matthew Hausknecht
Michael Bowling

title: Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents

venue: Journal of Artificial Intelligence Research,

authors:
Jarryd Martin
Suraj Narayanan Sasikumar
Tom Everitt
Marcus Hutter

title: Count-Based Exploration in Feature Space for Reinforcement Learning

venue: In IJCAI,

authors:
André F T Martins
Ramón Fernandez Astudillo

title: From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

authors:
Piotr Mirowski
Razvan Pascanu
Fabio Viola
Hubert Soyer
Andrew J Ballard
Andrea Banino
Misha Denil
Ross Goroshin
Laurent Sifre
Koray Kavukcuoglu
Dharshan Kumaran
Raia Hadsell

title: Learning to Navigate in Complex Environments

authors:
Piotr Mirowski
Matthew Koichi Grimes
Mateusz Malinowski
Karl Moritz Hermann
Keith Anderson
Denis Teplyashin
Karen Simonyan
Koray Kavukcuoglu
Andrew Zisserman
Raia Hadsell

title: Learning to Navigate in Cities Without a Map

authors:
Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Andrei A. Rusu
Joel Veness
Marc G. Bellemare
Alex Graves
Martin Riedmiller
Andreas K. Fidjeland
Georg Ostrovski
Stig Petersen
Charles Beattie
Amir Sadik
Ioannis Antonoglou
Helen King
Dharshan Kumaran
Daan Wierstra
Shane Legg
Demis Hassabis

title: Human-level control through deep reinforcement learning

venue: Nature,

authors:
Volodymyr Mnih
Adrià Puigdomènech Badia
Mehdi Mirza
Alex Graves
Timothy P Lillicrap
Tim Harley
David Silver
Koray Kavukcuoglu

title: Asynchronous Methods for Deep Reinforcement Learning

authors:
May-Britt Moser
David Rowland
Edvard I Moser

title: Place Cells, Grid Cells, and Memory

venue: Cold Spring Harbor perspectives in medicine,

authors:
Junhyuk Oh
Xiaoxiao Guo
Honglak Lee
Richard L Lewis
Satinder P Singh

title: Action-Conditional Video Prediction using Deep Networks in Atari Games

venue: In NIPS,

authors:
Ian Osband
Charles Blundell
Alexander Pritzel
Benjamin Van Roy

title: Deep Exploration via Bootstrapped DQN

authors:
Georg Ostrovski
Marc G. Bellemare
Aaron van den Oord
Remi Munos

title: Count-Based Exploration with Neural Density Models

authors:
Pierre-Yves Oudeyer
Frederic Kaplan

title: What is intrinsic motivation? A typology of computational approaches

venue: Frontiers in Neurorobotics,

authors:
Deepak Pathak
Pulkit Agrawal
Alexei A Efros
Trevor Darrell

title: Curiosity-driven Exploration by Self-supervised Prediction

authors:
Matthias Plappert
Rein Houthooft
Prafulla Dhariwal
Szymon Sidor
Richard Y Chen
Xi Chen
Tamim Asfour
Pieter Abbeel
Marcin Andrychowicz

title: Parameter Space Noise for Exploration

title: Objective criteria for the evaluation of clustering methods

venue: Journal of the American Statistical Association,

title: Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the World

venue: arXiv preprint arXiv:1804.06955,

title: Adaptive confidence and adaptive curiosity

authors:
John Schulman
Filip Wolski
Prafulla Dhariwal
Alec Radford
Oleg Klimov

title: Proximal policy optimization algorithms

venue: arXiv preprint arXiv:1707.06347,

authors:
Evan Shelhamer
Parsa Mahmoudieh
Max Argus
Trevor Darrell

title: Loss is its own Reward: Self-Supervision for Reinforcement Learning

venue: arXiv preprint arXiv:1612.07307,

authors:
Satinder Singh
Nuttapong Chentanez
Andrew G. Barto

title: Intrinsically Motivated Reinforcement Learning

authors:
Alexander L. Strehl
Michael L. Littman

title: An analysis of model-based interval estimation for markov decision processes

venue: Journal of Computer and System Sciences,

authors:
Haoran Tang
Rein Houthooft
Davis Foote
Adam Stooke
Xi Chen
Yan Duan
John Schulman
Filip De Turck
Pieter Abbeel

title: Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

title: The development and generalization of "contingency awareness" in early infancy: Some hypotheses

venue: Merrill-Palmer Quarterly of Behavior and Development,

authors:
Kelvin Xu
Jimmy Ba
Ryan Kiros
Kyunghyun Cho
Aaron C. Courville
Ruslan Salakhutdinov
Richard S Zemel
Yoshua Bengio

title: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

authors:
Zeyu Zheng
Junhyuk Oh
Satinder Singh

title: On Learning Intrinsic Rewards for Policy Gradient Methods

heading: 1 INTRODUCTION

text:
The success of reinforcement learning (RL) algorithms in complex environments hinges on the way they balance exploration and exploitation. There has been a surge of recent interest in developing effective exploration strategies for problems with high-dimensional state spaces and sparse rewards (Schmidhuber, 1991; Oudeyer & Kaplan, 2009; Houthooft et al., 2016; Bellemare et al., 2016; Osband et al., 2016; Pathak et al., 2017; Plappert et al., 2018; Zheng et al., 2018). Deep neural networks have seen great success as expressive function approximators within RL and as powerful representation learning methods for many domains. In addition, there have been recent studies on using neural network representations for exploration (Tang et al., 2017; Martin et al., 2017; Pathak et al., 2017). For example, count-based exploration with neural density estimation (Bellemare et al., 2016; Tang et al., 2017; Ostrovski et al., 2017) presents one of the state-of-the-art techniques on the most challenging Atari games with sparse rewards.
Despite the success of recent exploration methods, it is still an open question on how to construct an optimal representation for exploration. For example, the concept of visual similarity is used for learning density models as a basis for calculating pseudo-counts (Bellemare et al., 2016; Ostrovski et al., 2017). However, as Tang et al. (2017) noted, the ideal way to represent states should be based on what is relevant to solving the MDP, rather than only relying on visual similarity. In addition, there remains another question on whether the representations used for recent exploration works are easily interpretable. To address these questions, we investigate whether we can learn a complementary, more intuitive, and interpretable high-level abstraction that can be very effective in exploration by using the ideas of contingency awareness and controllable dynamics.
∗Equal contributions, listed in alphabetical order. †Now at DeepMind. 1Examples of the learned policy and the contingent regions are available at https://coex-rl.github.io/.
The key idea that we focus on in this work is the notion of contingency awareness (Watson, 1966; Bellemare et al., 2012) — the agent’s understanding of the environmental dynamics and recognizing that some aspects of the dynamics are under the agent’s control. Intuitively speaking, this can represent the segmentation mask of the agent operating in the 2D or 3D environments (yet one can think of more abstract and general state spaces). In this study, we investigate the concept of contingency awareness based on self-localization, i.e., the awareness of where the agent is located in the abstract state space. We are interested in discovering parts of the world that are directly dependent on the agent’s immediate action, which often reveal the agent’s approximate location.
For further motivation on the problem, we note that contingency awareness is a very important concept in neuroscience and psychology. In other words, being self-aware of one’s location is an important property within many observed intelligent organisms and systems. For example, recent breakthroughs in neuroscience, such as the Nobel Prize winning work on the grid cells (Moser et al., 2015; Banino et al., 2018), show that organisms that perform very well in spatially-challenging tasks are self-aware of their location. This allows rats to navigate, remember paths to previously visited places and important sub-goals, and find shortcuts. In addition, the notion of contingency awareness has been shown as an important factor in developmental psychology (Watson, 1966; Baeyens et al., 1990). We can think of self-localization (and more broadly self-awareness) as a principled and fundamental direction towards intelligent agents.
Based on these discussions, we hypothesize that contingency awareness can be a powerful mechanism for tackling exploration problems in reinforcement learning. We consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). For example, in the context of 2D Atari games, contingency-awareness roughly corresponds to understanding the notion of controllable entities (e.g., the player’s avatar), which Bellemare et al. (2012) refer to as contingent regions. More concretely, as shown in Figure 1, in the game FREEWAY, only the chicken sprite is under the agent’s control and not the multiple moving cars; therefore the chicken’s location should be an informative element for exploration (Bellemare et al., 2012; Pathak et al., 2017).
In this study, we also investigate whether contingency awareness can be learned without any external annotations or supervision. For this, we provide an instantiation of an algorithm for automatically learning such information and using it for improving exploration on a 2D ALE environment (Bellemare et al., 2013). Concretely, we employ an attentive dynamics model (ADM) to predict the agent’s action chosen between consecutive states. It allows us to approximate the agent’s position in 2D environments, but unlike other approaches such as (Bellemare et al., 2012), it does not require any additional supervision to do so. The ADM learns in an online and self-supervised fashion with pure observations as the agent’s policy is updated and does not require hand-crafted features, an environment simulator, or supervision labels for training.
In experimental evaluation, our methods significantly improve the performance of A2C on hardexploration Atari games in comparison with competitive methods such as density-based exploration (Bellemare et al., 2016; Ostrovski et al., 2017) and SimHash (Tang et al., 2017). We report very strong results on sparse-reward Atari games, including the state-of-the-art performance on the notoriously difficult MONTEZUMA’S REVENGE, when combining our proposed exploration strategy with PPO (Schulman et al., 2017), without using expert demonstrations, explicit high-level information (e.g., RAM states), or resetting the environment to an arbitrary state.
We summarize our contributions as follows:
• We demonstrate the importance of learning contingency awareness for efficient exploration in challenging sparse-reward RL problems.
• We develop a novel instance of attentive dynamics model using contingency and controllable dynamics to provide robust localization abilities across the most challenging Atari environments.
• We achieve a strong performance on difficult sparse-reward Atari games, including the state-ofthe-art score on the notoriously challenging MONTEZUMA’S REVENGE.
Overall, we believe that our experiments confirm the hypothesis that contingency awareness is an extremely powerful concept for tackling exploration problems in reinforcement learning, which opens up interesting research questions for further investigations.

heading: 2 RELATED WORK

text:
Self-Localization. The discovery of grid cells (Moser et al., 2015) motivates working on agents that are self-aware of their location. Banino et al. (2018) emphasize the importance of self-localization and train a neural network which learns a similar mechanism to grid cells to perform tasks related to spatial navigation. The presence of grid cells is correlated with high performance. Although grid cells seem tailored to 2D or 3D problems that animals encounter in their life, it is speculated that their use can be extended to more abstract spaces. A set of potential approaches to self-localization ranges from ideas specific to a given environment, e.g., SLAM (Durrant-Whyte & Bailey, 2006), to methods with potential generalizability (Mirowski et al., 2017; Jaderberg et al., 2017; Mirowski et al., 2018).
Self-supervised Dynamics Model and Controllable Dynamics. Several works have used forward and/or inverse dynamics models of the environment (Oh et al., 2015; Agrawal et al., 2016; Shelhamer et al., 2017). Pathak et al. (2017) employ a similar dynamics model to learn feature representations of states that captures controllable aspects of the environment. This dense representation is used to design a curiosity-driven intrinsic reward. The idea of learning representations on relevant aspects of the environment by learning auxiliary tasks is also explored in (Jaderberg et al., 2017; Bengio et al., 2017; Sawada, 2018). Our presented approach is different as we focus on explicitly discovering controllable aspects using an attention mechanism, resulting in better interpretability.
Exploration and Intrinsic Motivation. The idea of providing an exploration bonus reward depending on the state-action visit-count was proposed by Strehl & Littman (2008) (MBIE-EB), originally under a tabular setting. Later it has been combined with different techniques to deal with high-dimensional state spaces. Bellemare et al. (2016) use a Context-Tree Switching (CTS) density model to derive a state pseudo-count, whereas Ostrovski et al. (2017) use PixelCNN as a state density estimator. Martin et al. (2017) also construct a visitation density model over a compressed feature space rather than the raw observation space. Alternatively, Tang et al. (2017) propose a localitysensitive hashing (LSH) method to cluster states and maintain a state-visitation counter based on a form of similarity between frames. We train an agent with a similar count-based exploration bonus, but the way of maintaining state counter seems relatively simpler in that key feature information (i.e., controllable region) is explicitly extracted from the observation and directly used for counting states. Another popular family of exploration strategies in RL uses intrinsic motivation (Schmidhuber, 1991; Singh et al., 2004; Oudeyer & Kaplan, 2009; Barto, 2013). These methods encourage the agent to look for something surprising in the environment which motivates its search for novel states, such as surprise (Achiam & Sastry, 2017), curiosity (Pathak et al., 2017; Burda et al., 2018), and diversity (Eysenbach et al., 2018), or via feature control (Jaderberg et al., 2017; Dilokthanakul et al., 2017).

heading: 3 APPROACH

heading: 3.1 DISCOVERING CONTINGENCY VIA ATTENTIVE DYNAMICS MODEL

text:
To discover the region of the observation that is controllable by the agent, we develop an instance of attentive dynamics model (ADM) based on inverse dynamics finv. The model takes two consecutive input frames (observations) st−1, st ∈ S as input and aims to predict the action (at−1 ∈ A) taken by
the agent to transition from st−1 to st: ât−1 = finv(st−1, st). (1)
Our key intuition is that the inverse dynamics model should attend to the most relevant part of the observation, which is controllable by the agent, to be able to classify the actions. We determine whether each region in a H ×W grid is controllable, or in other words, useful for predicting the agent’s action, by using a spatial attention mechanism (Bahdanau et al., 2015; Xu et al., 2015). An overview of the model is shown in Figure 1.
Model. To perform action classification, we first compute a convolutional feature map φst = φ(st) ∈ RH×W×K based on the observation st using a convolutional neural network φ. We estimate a set of logit (score) vectors, denoted et(i, j) ∈ R|A|, for action classification from each grid cell (i, j) of the convolutional feature map. The local convolution features and feature differences for consecutive frames are fed into a shared multi-layer perceptron (MLP) to derive the logits as:
et(i, j) = MLP ([ φst (i, j)− φst−1(i, j); φst (i, j) ]) ∈ R|A|. (2)
We then compute an attention mask αt ∈ RH×W corresponding to frame t, which indicates the controllable parts of the observation st. Such attention masks are computed via a separate MLP from the features of each region (i, j), and then converted into a probability distribution using softmax or sparsemax operators (Martins & Astudillo, 2016):
αt = sparsemax(α̃t) where α̃t(i, j) = MLP ( φst (i, j) ) , (3)
so that ∑ i,j αt(i, j) = 1. The sparsemax operator is similar to softmax but yields a sparse attention, leading to more stable performance. Finally, the logits et(i, j) from all regions are linearly combined using the attention probabilities αt:
p(ât−1 | st−1, st) = softmax (∑ i,j αt(i, j) · et(i, j) ) ∈ R|A|. (4)
Training. The model can be optimized with the standard cross-entropy loss Laction(a∗t−1, ât−1) with respect to the ground-truth action a∗t−1 ∈ A that the agent actually has taken. Based on this formulation, the attention probability αt(i, j) should be high only on regions (i, j) that are predictive of the agent’s actions. Our formulation enables learning to localize controllable entities in a selfsupervised way without any additional supervisory signal, unlike some prior work (e.g., (Bellemare et al., 2012)) that adopts simulators to collect extra supervisory labels.
Optimizing the parameters of ADM on on-policy data is challenging for several reasons. First, the ground-truth action may be unpredictable for given pairs of frames, leading to noisy labels. For example, actions taken in uncontrollable situations do not have any effect (e.g., when the agent is in the middle of jumping in MONTEZUMA’S REVENGE). Second, since we train the ADM online along with the policy, the training examples are not independently and identically distributed, and the data distribution can shift dramatically over time. Third, the action distribution from the agent’s policy can run into a low entropy2, being biased towards certain actions. These issues may prevent the ADM from generalization to novel observations, which hurts exploration. Generally, we prefer models that quickly adapt to the policy and learn to localize the controllable regions in a robust manner.
To mitigate the aforementioned issues, we adopt a few additional objective functions. We encourage the attention distribution to attain a high entropy by including an attention entropy regularization loss, i.e., Lent = −H(αt). This term penalizes over-confident attention masks, making the attention closer to uniform whenever action prediction is not possible. We also train the logits corresponding to each grid cell independently using a separate cross-entropy loss: p(âi,jt−1 | et(i, j)) = softmax(et(i, j)). These additional cross-entropy losses, denoted Li,jcell, allow the model to learn from unseen observations even when attention fails to perform well at first. The entire training objective becomes:
LADM = Laction + ∑ i,j L i,j cell + λentLent (5)
where λent is a mixing hyperparameter.

heading: 3.2 COUNT-BASED EXPLORATION WITH CONTINGENT REGIONS

text:
One natural way to take advantage of discovered contingent regions for exploration is count-based exploration. The ADM can be used to localize the controllable entity (e.g., the agent’s avatar)
2We note that an entropy regularization term (e.g., Eq.(9)) is used when learning the policy.
from an observation st experienced by the agent. In 2D environments, a natural discretization (x, y) = argmax(j,i) αt(i, j) provides a good approximation of the agent’s location within the current observation3. This provides a key piece of information about the current state of the agent.
Inspired by previous work (Bellemare et al., 2016; Tang et al., 2017), we add an exploration bonus of r+ to the environment reward, where r+(s) = 1/ √ #(ψ(s)) and #(ψ(s)) denotes the visitation count of the (discrete) mapped state ψ(s), which consists of the contingent region (x, y). We want to find a policy π that maximizes the expected discounted sum of environment rewards rext plus count-based exploration rewards r+, denotedR = Eπ [∑ t γ t (β1r ext(st, at) + β2r +(st)) ] , where β1, β2 ≥ 0 are hyperparameters that balance the weight of environment reward and exploration bonus. For every state st encountered at time step t, we increase the counter value #(ψ(st)) by 1 during training. The full procedure is summarized in Algorithm 1 in Appendix A.

heading: 4 EXPERIMENTS

text:
In the experiments below we investigate the following key questions: • Does the contingency awareness in terms of self-localization provide a useful state abstraction
for exploration? • How well can the self-supervised model discover the ground-truth abstract states? • How well does the proposed exploration strategy perform against other exploration methods?

heading: 4.1 EXPERIMENTS WITH A2C

text:
We evaluate the proposed exploration strategy on several difficult exploration Atari 2600 games from the Arcade Learning Environment (ALE) (Bellemare et al., 2013). We focus on 8 Atari games including FREEWAY, FROSTBITE, HERO, PRIVATEEYE, MONTEZUMA’S REVENGE, QBERT, SEAQUEST, and VENTURE. In these games, an agent without an effective exploration strategy can often converge to a suboptimal policy. For example, as depicted in Figure 2, the Advantage Actor-Critic (A2C) baseline (Mnih et al., 2016) achieves a reward close to 0 on MONTEZUMA’S REVENGE, VENTURE, FREEWAY, FROSTBITE, and PRIVATEEYE, even after 100M steps of training. By contrast, our proposed technique, which augments A2C with count-based exploration with the location information learned by the attentive dynamics model, denoted A2C+CoEX (CoEX stands for “Contingency-aware Exploration”), significantly outperforms the A2C baseline on six out of the 8 games.
We compare our proposed A2C+CoEX technique against the following baselines:4
3To obtain more accurate localization by taking temporal correlation into account, we can use exponential smoothing as αt(i, j) = (1− ωt)αt−1(i, j) + ωtαt(i, j), where ωt = max(i,j){αt(i, j)}.
4In Section 4.6, we also report experiments using Proximal Policy Optimization (PPO) (Schulman et al., 2017) as a baseline, where our PPO+CoEX achieves the average score of >11,000 on MONTEZUMA’S REVENGE.
• A2C: an implementation adopted from OpenAI baselines (Dhariwal et al., 2017) using the default hyperparameters, which serves as the building block of our more complicated baselines. • A2C+Pixel-SimHash: Following (Tang et al., 2017), we map 52×52 gray-scale observations to 128-bit binary codes using random projection followed by quantization (Charikar, 2002). Then, we add a count-based exploration bonus based on quantized observations.
As a control experiment, we evaluate A2C+CoEX+RAM∗, our contingency-aware exploration method together with the ground-truth location information obtained from game’s RAM. It is roughly an upper-bound of the performance of our approach.

heading: 4.2 IMPLEMENTATION DETAILS

text:
For the A2C (Mnih et al., 2016) algorithm, we use 16 parallel actors to collect the agent’s experience, with 5-step rollout, which yields a minibatch of size 80 for on-policy transitions. We use the last 4 observation frames stacked as input, each of which is resized to 84× 84 and converted to grayscale as in (Mnih et al., 2015; 2016). We set the end of an episode to when the game ends, rather than when the agent loses a life. Each episode is initialized with a random number of no-ops (Mnih et al., 2015). More implementation details can be found in Appendix A and B.
For the ADM, we take observation frames of size 160×160 as input (resized from the raw observation of size 210× 160).5 We employ a 4-layer convolutional neural network that produces a feature map φ(st) with a spatial grid size of H ×W = 9× 9. As a result, the prediction of location coordinates lies in the 9× 9 grid. In some environments, the contingent regions within the visual observation alone are not sufficient to determine the exact location of the agent within the game; for example, the coordinate cannot
5In some games such as Venture, the agent is depicted in very small pixels, which might be hardly recognizable in rescaled 84× 84 images.
solely distinguish between different rooms in HERO, MONTEZUMA’S REVENGE, and PRIVATEEYE, etc. Therefore, we introduce a discrete context representation c ∈ Z that summarizes the high-level visual context in which the agent currently lies. We use a simple clustering method similar to (Kulis & Jordan, 2012), which we refer to as observation embedding clustering that clusters the random projection vectors of the input frames as in (Tang et al., 2017), so that different contexts are assigned to different clusters. We further explain this heuristic approach more in detail in Appendix D.
In sparse-reward problems, the act of collecting a reward is rare but frequently instrumental for the future states of the environment. The cumulative rewardRt = ∑t−1 t′=0 r
ext(st′ , at′) from the beginning of the episode up to the current step t, can provide a useful high-level behavioral context because collecting rewards can trigger significant changes to the agent’s state and as a result the optimal behavior can change as well. In this sense, the agent should revisit the previously visited location for exploration when the context changes. For example, in MONTEZUMA’S REVENGE, if the agent is in the first room and the cumulative reward is 0, we know the agent has not picked up the key and the optimal policy is to reach the key. However, if the cumulative reward in the first room is 100, it means the agent has picked up the key and the next optimal goal is to open a door and move on to the next room. Therefore, we could include the cumulative reward as a part of state abstraction for exploration, which leads to empirically better performance.
To sum up, for the purpose of count-based exploration, we utilize the location (x, y) of the controllable entity (i.e., the agent) in the current observation discovered by ADM (Section 3.1), a context representation c ∈ Z that denotes the high level visual context, and a cumulative environment reward R ∈ Z that represents the exploration behavioral state. In such setting, we may denote ψ(s) = (x, y, c, R).

heading: 4.3 PERFORMANCE OF COUNT-BASED EXPLORATION

text:
Figure 2 shows the learning curves of the proposed methods on 8 Atari games. The performance of our method A2C+CoEX and A2C+CoEX+RAM as well as the baselines A2C and A2C+PixelSimHash are summarized in Table 1. In order to find a balance between the environment reward and the exploration bonus reward, we perform a hyper-parameter search for the proper weight of the environment reward β1 and the exploration reward β2 for A2C+CoEX+RAM, as well as for A2C+CoEX. The hyper-parameters for the two ended up being the same, which is consistent with our results. For fair comparison, we also search for the proper weight of environment reward for A2C baseline. The best hyper-parameters for each game are shown in Table 5 in Appendix B.
Compared to the vanilla A2C, the proposed exploration strategy improves the score on all the hard-exploration games. As shown in Table 1, provided the representation (x, y, c, R) is perfect, A2C+CoEX+RAM achieves a significant improvement over A2C by encouraging the agent to visit novel locations, and could nearly solve these hard exploration games as training goes on.
Furthermore, A2C+CoEX using representations learned with our proposed attentive dynamics model and observation embedding clustering also outperforms the A2C baseline. Especially on FREEWAY, FROSTBITE, HERO, MONTEZUMA’S REVENGE, QBERT and SEAQUEST, the performance is comparable with A2C+CoEX+RAM, demonstrating the usefulness of the contigency-awareness information discovered by ADM.
Comparison to other count-based exploration methods. Table 2 compares the proposed method with previous state-of-the-art results, where our proposed method outperforms the other methods on 5 out of 8 games. DQN-PixelCNN is the strongest alternative achieving a state-of-the-art performance on some of the most difficult sparse-reward games. We argue that using Q-learning as the base learner with DQN-PixelCNN makes the direct comparison with A2C+CoEX not completely adequate. Note that the closest alternative count-based exploration method to A2C+CoEX would be A3C+ (Bellemare
et al., 2016), which augments A3C (Mnih et al., 2016) with exploration bonus derived from pseudocount, because A2C and A3C share a similar policy learning method. With that in mind, one can observe a clear improvement of A2C+CoEX over A3C+ on all of the 8 Atari games.

heading: 4.4 ANALYSIS OF ATTENTIVE DYNAMICS MODEL

text: We also analyze the performance of the ADM that learns the controllable dynamics of the environment. As a performance metric, we report the average distance between the ground-truth agent location (x∗, y∗) and the predicted location (x, y) within the 9×9 grid: ‖(x, y)−(x∗, y∗)‖2. The ground-truth location of the agent is extracted from RAM6, then rescaled so that the observation image frame fits into the 9× 9 grid. Figure 3 shows the results on 4 Atari games (MONTEZUMA’S REVENGE, SEAQUEST, HERO, and VENTURE). The ADM is able to quickly capture the location of the agent without any supervision of localization, despite the agent constantly visiting new places. Typically the predicted location is on average 1 or 2 grid cells away from the ground-truth location. Whenever a novel scene is encountered (e.g., the second room in MONTEZUMA’S REVENGE at around 10M steps), the average distance temporarily increases but quickly drops again as the model learns the new room. We provide videos of the agents playing and localization information as the supplementary material.7

heading: 4.5 ANALYSIS OF OBSERVATION EMBEDDING CLUSTERING

text:
To make the agent aware of a change in high-level visual context (i.e., rooms in Atari games) in some games such as MONTEZUMA’S REVENGE, VENTURE, HERO, and PRIVATEEYE, we obtain a representation of the high-level context and use it for exploration. The high-level visual contexts are different from each other (different layouts, objects, colors, etc.), so the embedding generated by a random projection is quite distinguishable and the clustering is accurate and robust.
For evaluation, given an observation in Atari games, we compare the discrete representation (i.e., which cluster it is assigned to) based on the embedding from random projection to the ground-truth room number extracted from RAM. The Adjusted Rand Index (ARI) (Rand, 1971) measures the similarity between these two data clusterings. The ARI may only yield a value between 0 and 1, and is exactly 1 when the clusterings are identical.
The curves of the Adjusted Rand Index are shown in Figure 4. For MONTEZUMA’S REVENGE and VENTURE, the discrete representation as room number is roughly as good as the ground-truth. For HERO and PRIVATEEYE, since there are many rooms quite similar to one another, it is more challenging to accurately cluster the embeddings. The samples shown in Figure 7 in Appendix D show reasonable performances of the clustering method on all these games.

heading: 4.6 ADDITIONAL EXPERIMENTS WITH PPO

text:
We also evaluate the proposed exploration algorithm on MONTEZUMA’S REVENGE using the sticky actions environment setup (Machado et al., 2017) identical to the setup found in (Burda et al., 2019). In the sticky action setup, the agent randomly repeats the previous action with probability of 0.25, preventing the algorithm from simply memorizing the correct sequence of actions and relying on determinism. The agent is trained with Proximal Policy Optimization (PPO) (Schulman et al., 2017) in conjunction with the proposed exploration method using 128 parallel actors to collect the experience. We used reward normalization and advantage normalization as in (Burda et al., 2018).
6Please note that the location from RAM is used only for analysis and evaluation purposes. 7A demo video of the learnt policy and localization is available at https://coex-rl.github.io/.
The method, denoted PPO+CoEX, achieves the score of 11,618 at 500M environment steps (2 billion frames) on MONTEZUMA’S REVENGE, when averaged over 3 runs. The learning curve is illustrated in Figure 5. Since the vanilla PPO baseline achieves a score near 0 (our runs) or 1,797 (Burda et al., 2019), this result is not solely due to the benefits of PPO. There is another approach "Exploration by Random Network Distillation" (Burda et al., 2019) concurrent with our work which achieves similar performance by following a slightly different philosophy.

heading: 4.7 DISCUSSIONS AND FUTURE WORK

text:
This paper investigates whether discovering controllable dynamics via an attentive dynamics model (ADM) can help exploration in challenging sparse-reward environments. We demonstrate the effectiveness of this approach by achieving significant improvements on notoriously difficult video games. That being said, we acknowledge that our approach has certain limitations. Our currently presented instance of state abstraction method mainly focuses on controllable dynamics and employs a simple clustering scheme to abstract away uncontrollable elements of the scene. In more general setting, one can imagine using attentive (forward or inverse) dynamics models to learn an effective and compact abstraction of the controllable and uncontrollable dynamics as well, but we leave this to future work.
Key elements of the current ADM method include the use of spatial attention and modelling of the dynamics. These ideas can be generalized by a set of attention-based dynamics models (ADM) operating in forward, inverse, or combined mode. Such models could use attention over a lowerdimensional embedding that corresponds to an intrinsic manifold structure from the environment (i.e., intuitively speaking, this also corresponds to being self-aware of (e.g., locating) where the agent is in the abstract state space). Our experiments with the inverse dynamics model suggest that the mechanism does not have to be perfectly precise, allowing for some error in practice. We speculate that mapping to such subspace could be obtained by techniques of embedding learning.
We note that RL environments with different visual characteristics may require different forms of attention-based techniques and properties of the model (e.g., partial observability). Even though this paper focuses on 2D video games, we believe that the presented high-level ideas of learning contingency-awareness (with attention and dynamics models) are more general and could be applicable to more complex 3D environments with some extension. We leave this as future work.

heading: 5 CONCLUSION

text: We proposed a method of providing contingency-awareness through an attentive dynamics model (ADM). It enables approximate self-localization for an RL agent in 2D environments (as a specific perspective). The agent is able to estimate its position in the space and therefore benefits from a compact and informative representation of the world. This idea combined with a variant of countbased exploration achieves strong results in various sparse-reward Atari games. Furthermore, we report state-of-the-art results of >11,000 points on the infamously challenging MONTEZUMA’S REVENGE without using expert demonstrations or supervision. Though in this work we focus mostly on 2D environments in the form of sparse-reward Atari games, we view our presented high-level concept and approach as a stepping stone towards more universal algorithms capable of similar abilities in various RL environments.

heading: A SUMMARY OF TRAINING ALGORITHM

text:
Algorithm 1 A2C+CoEX Initialize parameter θADM for attentive dynamics model fADM Initialize parameter θA2C for actor-critic network Initialize parameter θc for context embedding projector if applicable (which is not trainable) Initialize transition buffer E ← ∅ for each iteration do B Collect on-policy transition samples, distributed over K parallel actors for each step t do st ← Observe state at ∼ πθ(at|st) st+1, r ext t ← Perform action at in the environment
B Compute the contingent region information αt+1 ← Compute the attention map of st+1 using fADM c(st+1)← Compute the observation embedding cluster of st+1 (Algorithm 2) B Increment state visitation counter based on the representation ψ(st+1)← (argmax(i,j) αt+1(i, j), c(st+1), b ∑t k=0 r ext k c) #(ψ(st+1))← #(ψ(st+1)) + 1 r+t ← 1√#(ψ(st+1)) Store transition E ← E ∪ { (st, at, st+1, β1clip(r ext t ,−1, 1) + β2r+t )
} end for B Perform actor-critic using on-policy samples in E θA2C ← θA2C − η∇θA2CLA2C B Train the attentive dynamics model using on-policy samples in E θADM ← θADM − η∇θADMLADM Clear transition buffer E ← ∅
end for
The learning objective LADM is from Equation (5). The objective LA2C of Advantage Actor-Critic (A2C) is as in (Mnih et al., 2016; Dhariwal et al., 2017):
LA2C = E(s,a,r)∼E [ LA2Cpolicy + 1
2 LA2Cvalue
] (6)
LA2Cpolicy = − log πθ(at|st)(Rnt − Vθ(st))− αHt(πθ) (7)
LA2Cvalue = 1
2
( Vθ(st)−Rnt )2 (8)
Ht(πθ) = − ∑ a πθ(a|st) log πθ(a|st) (9)
where Rnt = ∑n−1 i=0 γ irt+i + γ nVθ(st+n) is the n-step bootstrapped return and α is a weight for the standard entropy regularization loss termHt(πθ). We omit the subscript as θ = θA2C when it is clear.

heading: B ARCHITECTURE AND HYPERPARAMETER DETAILS

text: The architecture details of the attentive dynamics model (ADM), the policy network, and hyperparameters are as follows.

heading: C EXPERIMENT WITH RAM INFORMATION

text: In order to understand the performance of exploration with perfect representation, we extract the ground-truth location of the agent and the room number from RAM, and then run count-based exploration with the perfect (x, y, c, R). Figure 6 shows the learning curves of the experiments; we could see A2C+CoEX+RAM acts as an upper bound performance of our proposed method.

heading: D OBSERVATION EMBEDDING CLUSTERING

text:
We describe the detail of a method to obtain the observation embedding. Given an observation of shape (84, 84, 3), we flatten the observation and project it to an embedding of dimension 64. We randomly initialize the parameter of the fully-connected layer for projection, and keep the values unchanged during the training to make the embedding stationary.
For the embedding of these observations, we cluster them based on a threshold value τ . The value of τ for each game with change of rooms is listed in Table 5. If the distance between the current embedding and the center mean(c) of a cluster c is less than the threshold, we assign this embedding to the cluster with the smallest distance and update its center with the mean value of all embeddings belonging to this cluster. If the distance between the current embedding and the center of any cluster is larger than the threshold, we create a new cluster and this embedding is assigned to this new cluster.
Algorithm 2 Observation Embedding Clustering Initialize parameter θc for context embedding projector if applicable (which is not trainable) Initialize threshold τ for clustering Initialize clusters set C ← ∅ for each observation s do B Get embedding of the observation from the random projection v ← fθc(s) B Find a cluster to which the current embedding fits, if any Find a cluster c ∈ C with smallest ‖mean(c)− v‖ ≤ τ , or NIL if there is no such if c 6= NIL then c← c ∪ v
else B if there’s no existing cluster that v should be assigned to, create a new one C ← C ∪ {v}
end if end for
In Figure 7, we also show the samples of observation in each cluster. We could see observations from the same room are assigned to the same cluster and different clusters correspond to different rooms.

heading: E ABLATION STUDY ON ATTENTIVE DYNAMICS MODEL

text:
We conduct a simple ablation study on the learning objectives of ADM, described in Equation (5). We evaluate the performance of ADM when trained on the same trajectory data under different combinations of loss terms, simulating batches of on-policy transition data to be replayed. The sample trajectory was obtained from an instance of A2C+CoEX+RAM and kept same across all the runs, which allows a fair comparison between different variants. We compare the following four methods:
• ADM (action) : train ADM using Laction only • ADM (action, cell) : train ADM using Laction and Lcell • ADM (action, ent) : train ADM using Laction and Lent • ADM (action, cell, ent) : train ADM using all losses (Laction,Lcell,Lent)
Figure 8 shows the average distance between the ground-truth location of the agent and the predicted one by ADM during the early stages of training. On MONTEZUMA’S REVENGE, there is only little difference between the variants although the full model worked slightly better on average. On FREEWAY, the effect of loss terms is more clear; in the beginning the agent tends to behave suboptimally by taking mostly single actions only (UP out of three action choices — UP, DOWN, and NO-OP), hence very low entropyH(π(·|s)), which can confuse the ADM of telling which part is actually controllable as the action classifier would give correct answer regardless of attention. We can observe additional loss terms help the model quickly correct the attention to localize the controllable object among the uncontrollable clutters with better stability.
In another ablation study, we compare the end performance of the A2C+CoEX agent with the ADM jointly trained under different loss objectives on these three games (MONTEZUMA’S REVENGE, FREEWAY and SEAQUEST). In our experiments, the variant with full ADM worked best on MONTEZUMA’S REVENGE and FREEWAY. The minimal training objective of ADM (i.e., Laction) also solely works reasonably well, but with the combination of other loss terms we can attain a more stable performance.

heading: F ABLATION STUDY ON THE STATE REPRESENTATION

text:
We present a result of additional ablation study on the state representation ψ(s) used in count-based exploration. The following baselines are considered:
• A2C+CoEX(c): Uses only the context embedding for exploration, i.e., ψ(s) = (c). • A2C+CoEX(c,R): Uses only the context embedding and the cumulative reward for exploration
without contingent region information, i.e., ψ(s) = (c,R). • A2C+CoEX(x, y, c): Uses the contingent region information (x, y) as well as the context
embedding c, however without the cumulative reward component, i.e., ψ(s) = (x, y, c).
One can also consider another baseline similar to A2C+CoEX(c,R) with ψ(s) = (x, y, c, R), where the location information (x, y) is replaced with random coordinates uniformly sampled from the grid. It ablates the learned contingent regions. However, we found that it performs similarly to the presented A2C+CoEX(c,R) baseline.
The experimental results are summarized in Table 6 and Figure 10. The variants without contingent regions (i.e., A2C+CoEX(c) and A2C+CoEX(c,R) performed significantly worse in most of the games than A2C+CoEX(x, y, c) and A2C+CoEX(x, y, c, R) giving little improvement over the A2C baseline. Most notably, in the games with the hardest exploration such as MONTEZUMA’S REVENGE and VENTURE, the performance is hardly better than the vanilla A2C or a random policy, achieving a score as low as zero. The variants with contingent region information worked best and comparable to each other. We observe that using the cumulative reward (total score) for exploration gives a slight improvement on some environments. These results support the effectiveness of the learned contingency-awareness information in count-based exploration.


summaries: This works propose a new approach to learn to sample (or generate) the parameters of a deep neural networks to solve a task. They propose a new architecture inspired by hyper networks and adversarial auto-encoders, where the parameters of the networks are generated from a low dimensional latent space. By using an ensemble of networks sampled with their approach they're able to get state of the art results on uncertainty estimation.

summaries: This paper proposes a technique for learning a distribution over parameters of a neural network such that samples from the distribution correspond to performant networks. The approach effectively encourages sampled parameters to have low loss on the training set, and also uses an adversarial loss to encourage the distribution of parameters to be Gaussian distributed. This approach can improve performance slightly by using ensembling and can be useful for uncertainty estimates for out-of-distribution examples. The approach is tested on a few simple problems and is shown to work well.

abstractText: We introduce HyperGAN, a generative network that learns to generate all the weight parameters of deep neural networks. HyperGAN first transforms low dimensional noise into a latent space, which can be sampled from to obtain diverse, performant sets of parameters for a target architecture. We utilize an architecture that bears resemblance to generative adversarial networks, but we evaluate the likelihood of samples with a classification loss. This is equivalent to minimizing the KL-divergence between the generated network parameter distribution and an unknown true parameter distribution. We apply HyperGAN to classification, showing that HyperGAN can learn to generate parameters which solve the MNIST and CIFAR-10 datasets with competitive performance to fully supervised learning, while learning a rich distribution of effective parameters. We also show that HyperGAN can also provide better uncertainty than standard ensembles. This is evaluated by the ability of HyperGAN-generated ensembles to detect out of distribution data as well as adversarial examples. We see that in addition to being highly accurate on inlier data, HyperGAN can provide reasonable uncertainty estimates.

authors:
Jimmy Ba
Geoffrey E Hinton
Volodymyr Mnih
Joel Z Leibo
Catalin Ionescu

title: Using fast weights to attend to the recent past

venue: In Advances in Neural Information Processing Systems,

authors:
Andreas Damianou
Neil Lawrence

title: Deep Gaussian processes

venue: Proceedings of the Sixteenth International Workshop on Artificial Intelligence and Statistics (AISTATS),

authors:
Yinpeng Dong
Fangzhou Liao
Tianyu Pang
Xiaolin Hu
Jun Zhu

title: Discovering adversarial examples with momentum

authors:
C.D. Freeman
J. Bruna

title: Topology and Geometry of Half-Rectified Network Optimization

venue: ArXiv e-prints,

authors:
Yarin Gal
Zoubin Ghahramani

title: Dropout as a bayesian approximation: Representing model uncertainty in deep learning

venue: In international conference on machine learning,

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Ian Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

venue: In International Conference on Learning Representations,

authors:
José Miguel Hernández-Lobato
Ryan Adams

title: Probabilistic backpropagation for scalable learning of bayesian neural networks

venue: In International Conference on Machine Learning,

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: Gans trained by a two time-scale update rule converge to a local nash equilibrium

authors:
Geoffrey E Hinton
David C Plaut

title: Using fast weights to deblur old memories

venue: In Proceedings of the ninth annual conference of the Cognitive Science Society,

authors:
Max Jaderberg
Karen Simonyan
Andrew Zisserman

title: Spatial transformer networks. In Advances in neural information processing

authors:
Xu Jia
Bert De Brabandere
Tinne Tuytelaars
Luc V Gool

title: Dynamic filter networks

authors:
Balaji Lakshminarayanan
Alexander Pritzel
Charles Blundell

title: Simple and scalable predictive uncertainty estimation using deep ensembles

authors:
Jonathan Lorraine
David Duvenaud

title: Stochastic hyperparameter optimization through hypernetworks

authors:
Christos Louizos
Max Welling

title: Multiplicative normalizing flows for variational bayesian neural networks

authors:
Richard Maclin
David W. Opitz

title: Popular ensemble methods: An empirical study

authors:
A. Madry
A. Makelov
L. Schmidt
D. Tsipras
A. Vladu

title: Towards Deep Learning Models Resistant to Adversarial Attacks

authors:
Nick Pawlowski
Andrew Brock
Matthew CH Lee
Martin Rajchl
Ben Glocker

title: Implicit weight uncertainty in neural networks

venue: arXiv preprint arXiv:1711.01297,

authors:
Tim Salimans
Ian Goodfellow
Wojciech Zaremba
Vicki Cheung
Alec Radford
Xi Chen

title: Improved techniques for training gans

title: Optimal transport: old and new, volume 338

venue: Springer Science & Business Media,

heading: 1 INTRODUCTION

text:
Since the inception of deep neural networks, it has been found that it is possible to train from different random initializations and obtain networks that, albeit having quite different parameters, achieve quite similar accuracy (Freeman & Bruna, 2016). It has further been found that ensembles of deep networks that are trained in such a way have significant performance advantages over single models (Maclin & Opitz, 2011), similar to the classical bagging approach in statistics. Ensemble models also have other benefits, such as being robust to outliers and being able to provide variance or uncertainty estimates over their inputs (Lakshminarayanan et al., 2017).
In Bayesian deep learning, there is a significant interest in having a probabilistic interpretation of network parameters and modeling a distribution over them. Earlier approaches mostly utilize dropout as a Bayesian approximation, by randomly setting different parameters to zero and thus integrating over many possible networks. (Gal & Ghahramani, 2016) showed that networks with dropout following each layer are equivalent to a deep Gaussian process (Damianou & Lawrence, 2013) marginalized over its covariance functions. They proposed MCdropout as a simple way to estimate model uncertainty. These approximations are not well aligned with current training patterns of neural networks. Applying dropout to every layer results in over-regularization and underfitting of the target function. Moreover, dropout does not integrate over the full variation of possible models, only those which may be reached from one (random) initialization.
As another interesting direction, hypernetworks (Ha et al., 2016) are neural networks which output parameters for a target neural network. The hypernetwork and the target network together form a single model which is trained jointly. The original hypernetwork produced the target weights as a deterministic function of its own weights, but Bayesian Hypernetworks (BHNs) (Krueger et al., 2017), and Multiplicative Normalizing Flows (MNF) (Louizos & Welling, 2016) generate model parameters by sampling a Gaussian prior. However, these approaches use normalizing flows to transform a simple prior into a sample of the more complicated posterior, which are composed of only bijective, invertible functions. This limits their scalability and the variety of learnable functions.
In this paper we explore an approach which focuses on generating all the parameters of a neural network, without assuming any fixed noise models on parameters. To keep our method scalable, we avoid utilizing invertible functions as in normalizing flow approaches, and instead utilize the ideas from generative adversarial networks (GANs). We especially observe recent Wasserstein Autoencoder (Tolstikhin et al., 2017) approaches. These approaches have demonstrated an impressive capability to model complicated, multimodal distributions. In our approach, a random noise vector is first encoded to a number of different random vectors, then each random vector is used to generate all parameters within one layer of a deep network. The generator is then trained with conventional maximum likelihood (classification/regression) on the parameters it generates, and an adversarial regularization keeps it from collapsing onto only one mode. In this way, it is possible to generate much larger networks than the dimensionality of the latent code, making our approach capable of generating all the parameters of a deep network with a single GPU. As an example, in our experiments on CIFAR-10 we start from a 256-dimensional latent vector and generate all 50, 000+ parameters in one pass, consuming only 4GB GPU memory. This shows that deep networks may indeed span a low-dimensional manifold, and could spur further thoughts and research.

heading: 1.1 SUMMARY OF CONTRIBUTIONS

text:
We propose HyperGAN, a novel approach for generating all the parameters for a target network architecture using a modified GAN, and we do so starting from a small Gaussian noise vector which scales well with the size of the output. Our approach is different from strict Bayesian approaches since we do not attempt to model the entire posterior, i.e., from an observed network we cannot compute its probability of it being generated. Moreover, we use deterministic generators, our only source of stochasticity is the initial prior sample. After our GAN is trained, one can directly generate many diverse, well-trained deep models without needing to further train or fine-tune them. The diversity of the models we can generate is beyond just adding dropout or scaling factors, which is shown by the superior performance of ensembles of the generated networks.
We believe HyperGAN is widely applicable to a variety of tasks. One area where populations of diverse networks show promise is in uncertainty estimation and anomaly detection. We show through a variety of experiments that populations of networks sampled from HyperGAN are able to approximate the data distribution such that it can detect out of distribution samples. We show that we can provide a reasonable measure of uncertainty by calculating the entropy within the predictive distribution of sampled networks. Our method is straightforward, as well as easy to train and sample from. We hope that we can inspire future work in estimation of the manifold of neural networks.

heading: 2 RELATED WORK

text:
Generating parameters for neural networks has been framed in contexts other than the Bayesian approaches described above. The hypernetwork framework (Ha et al., 2016) has come to describe models where one network directly supervises the weight updates of another network. (Pawlowski et al., 2017) uses hypernetworks to generate layer-wise weights for a target architecture given some auxiliary noise as input. The auxiliary noise is noted to be independent between layers, while we impose structure in our inputs to create diversity in our generated samples.
Computer vision methods often use data driven approaches as seen in methods such as Spatial Transformer networks (Jaderberg et al., 2015), or Dynamic Filter networks (Jia et al., 2016). In these methods the filter parameters of the main network are conditioned on the input data, receiving contextual scale and shift updates from an auxiliary network. Our method, however, generates parameters for the whole network. Furthermore our predicted parameters are highly nonlinear functions of the input, instead of simple affine transformations based on the input examples.
Recently, (Lakshminarayanan et al., 2017) proposed Deep Ensembles, where adversarial training was applied to standard ensembles to smooth the predictive variance. However, adversarial training is an expensive training process. Adversarial examples are generated for each batch of data seen. We seek a method to learn a distribution over parameters which does not require adversarial training.
Meta learning approaches use different kinds of weights in order to increase the generalization ability of neural networks. The first proposed method is fast weights (Hinton & Plaut, 1987) which uses an auxiliary (slow) network to produce weight changes in the target (fast) network, acting as a
short term memory store. Meta Networks (Munkhdalai & Yu, 2017) build on this approach by using an external neural memory store in addition to multiple sets of fast and slow weights. (Ba et al., 2016) augment recurrent networks with fast weights as a more biologically plausible memory system than Neural Turing Machines. Unfortunately, the generation of each predicting network requires querying the base (slow) learner many times. Many of the methods presented there, along with hyperparameter learning (Lorraine & Duvenaud, 2018), and the original hypernetwork, propose learning target weights which are deterministic functions of the training data. Our method instead captures a distribution over parameters, and provides a cheap way to directly sample full networks from a learned distribution.

heading: 3 HYPERGAN

text:
Taking note from the original hypernetwork framework for generating neural networks (Ha et al., 2016), we coin our approach HyperGAN. Standard GAN training dictates that we train a generator G to model the target distribution, aided by a discriminator network D. This entails acquiring a large dataset of trained neural network parameters, and providing samples to D to judge the output of the generator. However, a large collection of neural networks would be extremely costly to build. Instead, our data distribution is built online by evaluating the performance of our generated parameters at each step.
We begin with assuming that a neural network F(x; θ) with input x and parameters θ consisting of a given architecture withN layers, and a training setM with inputs and targets (X,Y ) = {xi, yi}Mi=1. The standard training regime consists of computing a loss function L(F(x; θ), y) and updating the parameters θ with backpropagation untilL is minimized. This works if we only want a point estimate of θ. However, if we want to generate more than one non-trivial network, some diversity is needed. Our approach to creating diversity is similar to a regular GAN architecture in that we start by drawing a random sample s ∼ S = N (0, Ij), where 0 is an all-zero vector and Ij is a j × j identity matrix, and generate parameters using a generator from s.
Figure 1 shows the HyperGAN architecture. Distinct from the standard GAN, we use parallel, untied generators to form the parameters of each layer. In order for that to work, we observe that parameters between network layers are not independent, but are strongly correlated. The parameters of each layer depend on the output and the parameters of the preceding layers. Therefore, the generated parameters must be correlated to produce well-performing neural networks. We propose to add a Mixer Q which maps s ∼ S to the mixed latent space Z ∈ RNd, d < j. An Nd-dimensional Q(s) is then partitioned into N layer embeddings [q1, . . . , qN ], each being a d-dimensional vector. Finally, we use N parallel generators G = {G1(q1) . . . GN (qn)} to generate the parameters θG for all layers in F . After generation, we can evaluate the new modelF(x, θG) on the training set. We define an objective which minimizes the error of generated parameters with respect to a task loss L:
inf G,Q
Es∼SE(x,y)∼(X,Y ) [L(F(x;G(Q(s))), y)] (1)
At each training step we generate a different network G(Q(s)) from a random s ∼ S, and then evaluate the loss function on a mini-batch from the training set. The resulting loss is backpropagated through the generators until θG minimizes the target loss L. The formulation in (1) presents a problem: the codes sampled from Q(s) will certainly collapse to the maximum likelihood (ML) estimate (when L is a log-likelihood). This means that the generators will learn a very narrow approximation of Θ, and indeed we see this happen in tables 3 and 4. To assure that the parameters are well distributed, we added an adversarial constraint on the mixed latent spaceD(Q(s)) so that it should not deviate too much from a Gaussian prior P . This constraint makes it closer to the generated parameters and ensures that Q(s) itself does not collapse to always outputting the same latent code. With this we arrive at the HyperGAN objective:
inf G,Q
Es∼SE(x,y)∼(X,Y ) [L(F(x;G(Q(s))), y)]− βD(Q(s),P) (2)
Where β is a hyperparameter, andD is the regularization term which penalizes the distance between the prior and the distribution of latent codes. In practice D could be any distance function between two distributions. We choose to parameterize D as a discriminator network D that output probabilities, and use the adversarial loss (Goodfellow et al., 2014) to approximate D(Q(s),P). Note that while P and S are both multivariate Gaussians, they are distinct distributions as seen in figure 1.
D := − N∑ i=1 (logD(pi) + log(1−D(qi))) (3)
Note that we find it difficult to learn a discriminator in the output (parameter) space because the dimensionality is high and there is no structure in those parameters to be utilized as in images (where CNNs can be trained). Our experiments show that regularizing in the latent space works well. We hypothesize this is because of overparametrization in θ. The latent space initializes with random projections, which have a restricted isometry property according to the Johnson-Lindenstrauss lemma, hence if the mixed latent factor q is Gaussian, a random projection from it preserves the distance and diversity from q to θ. If θ is indeed severely overparametrized, and it is possible to generate diverse parameters by maximizing the likelihood, then the generators would not collapse to a single θ, since that would require breaking the restricted isometry from the initialization.
This framework is general and can be adapted to a variety of tasks and losses. In this work, we show that HyperGAN can operate in both classification and regression settings. For multi-class classification, the generators and mixer are trained with the cross entropy loss function:
LH = M−1 M∑ i=1 yi log ( F(xi; θ) ) where θ = {G1(q1), . . . , Gn(qn)} (4)
For regression tasks we replace the cross entropy term with the mean squared error function:
Lmse = M−1 M∑ i=1 (yi −F(xi; θ))2 where θ = {G1(q1), . . . , Gn(qn)} (5)

heading: 3.1 LEARNING WITHOUT AN EXPLICIT TARGET DISTRIBUTION

text:
In implicit generative models such as GAN or WAE (Tolstikhin et al., 2017), it is always necessary to have a collection of inlier data points to train with. Inlier samples come from the distribution that is being estimated, and learning without them is difficult. HyperGAN does not have a given set of inlier points to train with. Instead, HyperGAN uses the error on the target loss to estimate the distance between the generated samples θG ∈ ΘG and the target samples θ ∈ Θ. We note that θG represents the maximum likelihood estimate of F(x; θ). (6) shows that by minimizing the error of the ML estimate on the log-likelihood, we are in fact minimizing the KL divergence between the target distribution Θ and the generated samples θG ∈ ΘG.
inf θG DKL(P (x|θ)||P (x|θG)) = inf θG EP (x|θG) [logP (x|θ)− logP (x|θG)]
= inf θG EP (x|θG) [− logP (x|θG)] (6)
Where the term logP (x|θ) is the entropy of the target distribution and does not contribute to the optimization problem. The derived likelihood function is flexible and used widely across different domains. In our work it is represented by the cross entropy and MSE losses that we study.
Its useful to note here how HyperGAN is distinct from other density estimators such as GAN and WAE. These methods use a likelihood function, such as MSE, to estimate the quality of the samples (in GANs the likelihood function is learned via the discriminator). The likelihood is given as the distance from generated samples, to given samples of the target distribution. HyperGAN instead estimates both the target and the approximation. We assume the target distribution exists, and update our approximation ΘG to better match Θ. If we have samples of the target distribution, we can reduce HyperGAN to a GAN (by moving the discriminator to the output space).We thoroughly explore the connections and differences to GANs and WAE in A.5. HyperGAN is different from fully probabilistic approaches such as MNF (Louizos & Welling, 2016) since one cannot compute the probability of the generated θ, nor can one encode a generated θ back to the latent spaces.

heading: 4 EXPERIMENTS

heading: 4.1 HIGH LEVEL DESCRIPTION AND EXPERIMENTAL SETUP

text:
We conduct a variety of experiments to show HyperGAN’s ability to achieve both high accuracy and obtain accurate uncertainty estimates. First we show classification performance on both MNIST and CIFAR-10 datasets. Next we examine HyperGAN’s ability to learn the variance of a simple 1D dataset. We perform experiments on anomaly detection: testing HyperGAN on notMNIST, and 5 classes of CIFAR-10 which are hidden during training. We also examine adversarial examples as extreme cases of off-manifold data, and test our robustness to them. In our experiments we compare with (Louizos & Welling, 2016) (MNF) as well as standard ensembles.
In all experiments we report results with two HyperGANs, one trained on MNIST and another on CIFAR-10. Both of our models take a 256 dimensional sample of S as input, but have different sized mixed latent spaces. The HyperGAN for the MNIST experiments consists of three weight generators, each using a 128 dimensional latent point as input. The target network for the MNIST experiments is a small two layer convolutional network, using leaky ReLU activations and 2x2 max pooling after each convolutional layer. Our HyperGAN trained on CIFAR-10 used 5 weight generators and latent points with dimensonality 256. The target architecture for CIFAR-10 tests consists of three convolutional layers, each followed by leaky ReLU and 2x2 max pooling. The exact architectures we used for the target networks is given in A.4. In our experiments we used the same network architecture across different methods. Note that our architecture is different from the LeNet-5 used in MNF, yet we see similar results as reported in (Louizos & Welling, 2016).

heading: HYPERGAN DETAILS

text:
The mixing network, generators, and discriminator are each parameterized by 2 layer MLPs with 512 units each and ReLU nonlinearities. We found in a pilot study that larger networks offered little performance benefit, and ultimately hurt scalability. In all experiments, we pretrain the mixer so that the mean and covariance ofQ(q|s) are close to P . It should be noted that the architecture is flexible. The number and width of layers may be varied without harming HyperGANs ability to model the target distribution. We trained our HyperGAN on MNIST using less than 1.5GB of memory on a single GPU, while CIFAR-10 used just 4GB, making HyperGAN surprisingly scalable.
In Table 1 we show some statistics of the networks generated by HyperGAN on MNIST. We note that HyperGAN can generate very diverse networks, as the variance of network parameters generated by the HyperGAN is significantly higher than standard training from different random initializations.

heading: 4.2 CLASSIFICATION

text:
First we evaluate the classification accuracy of HyperGAN on MNIST and CIFAR-10. Classification serves as an entrance exam into our other experiments, as the distribution we want to learn is over parameters which can effectively solve the classification task. We test with both single network samples, and ensembles. For our ensembles we average predictions from N sampled models with
the scoring rule p(y|x) = 1N
∑N
n=1 pn(y |x, θn). It should be noted that we did not perform fine tuning, or any additional training on the sampled networks. The results are shown in Table 2. We generate ensembles of different sizes and compare against both Bayesian (Louizos & Welling, 2016) (Krueger et al., 2017) and non-Bayesian (Lakshminarayanan et al., 2017) methods, as well as MC dropout (Gal & Ghahramani, 2016). We outperform all other methods by using a 100 network ensemble, across all datasets. For all other methods, we report the mean of 100 trials.

heading: 4.3 1-D TOY REGRESSION TASK

text:
We next evaluate the ability of HyperGAN to fit a simple 1D function from noisy samples. This dataset was first proposed by (Hernández-Lobato & Adams, 2015), and consists of a training set of 20 points drawn uniformly from the interval [−4, 4]. The targets are given by y = x3 + where ∼ N (0, 32). We used the same target architecture as in (Hernández-Lobato & Adams, 2015) Lakshminarayanan et al. (2017) and (Louizos & Welling, 2016): a one layer neural network with 100 hidden units and ReLU nonlinearity. For HyperGAN we use two layer generators, and 128 hidden units across all networks. Because this is a small task, we use only a 64 dimensional latent space. MSE loss is used as our target loss function to train HyperGAN.
Results in figure 2 show that HyperGAN clearly learns the target function and captures the variation in the data well. In addition, it can be seen that sampling more networks to compose a larger ensemble improves predictive uncertainty as we sample farther from the mean of the training data.

heading: 4.4 ANOMALY DETECTION

text:
To test our uncertainty measurements, we perform the same experiments as (Louizos & Welling, 2016) (MNF), (Lakshminarayanan et al., 2017); we measure the total entropy in predictions from HyperGAN-generated networks. We compare our uncertainty measurements with those of MNF and standard ensembles. For MNIST experiments we train a HyperGAN on the MNIST dataset, and test on the notMNIST dataset, which consists of 28x28 binary images of alphabetic letters. In this setting, we want the softmax probabilities on inlier MNIST examples to have maximum entropy - a single large activation close to 1. On off-manifold data we want to have equal probability across predictions. We test our CIFAR-10 model by training on the first 5 classes, and we use the latter
5 classes as out of distribution examples. To build an estimate of the predictive entropy we sample multiple networks from HyperGAN per example, and measure their predictive entropy..
In Fig. 3 we show that HyperGAN can separate CIFAR-10 inlier and outlier samples much better than MNF or standard ensembles. HyperGAN is less certain about data it does not recognize, as the probability of a low entropy prediction is overall lower on outliers. On notMNIST we also show separation, though HyperGAN is also overall less confidant about inliers then MNF. Conventionally trained ensembles without the HyperGAN, referred to as L2 networks in the figure, are highly overconfident on outliers and cannot provide a notion of uncertainty.

heading: 4.5 ADVERSARIAL DETECTION

text:
We employ the same experimental setup to the detection of adversarial examples, an extreme sort of off-manifold data. Adversarial examples are often optimized to lie within a small neighborhood of a classifier’s decision boundaries. They are created by adding perturbations in the direction of the greatest loss with respect to the model’s parameters. Because HyperGAN learns a distribution over parameters, it should be more robust to attacks. We generate adversarial examples using the Fast Gradient Sign method (FGSM) (Goodfellow et al., 2015) and Projected Gradient Descent (PGD) (Madry et al., 2017). FGSM adds a small perturbation to the target image in the direction of greatest loss. FGSM is known to underfit to the target model, hence it may transfer well across many similar models. In contrast, PGD takes many steps in the direction of greatest loss, producing a stronger adversarial example, at the risk of overfitting to a single set of parameters. This poses the following challenge: to detect attacks by FGSM and PGD, HyperGAN will need to generate diverse parameters to avoid both attacks. To detect adversarial examples, we first hypothesize that a single adversarial example will not fool the entire space of parameters learned by HyperGAN. If we then evaluate adversarial examples against many generated networks, then we should see a high entropy among predictions (softmax probabilities) for any individual class.
Adversarial examples have been shown to successfully fool ensembles (Dong et al., 2017), but with HyperGAN one can always generate significantly more models that can be added to the ensemble for the cost of one forward pass, making it hard to attack against. In Fig. 4 we test HyperGAN against adversarial examples generated to fool one network. It is shown that while those examples can fool 50%− 70% of the networks generated by HyperGAN, they usually do not fool all of them. We compare the performance of HyperGAN with ensembles of N ∈ {5, 10} models trained on MNIST with normal supervised training. We fuse their logits (unnormalized log probabilities) together as l(x) = ∑N n=1 wnln(x) where wn is the nth model weighting, and ln is the logits of the nth model. In all experiments we consider uniformly weighted ensembles. For HyperGAN we simply sample the generators to create as many models as we need, then we fuse their logits together. Specifically we test ensembles with N ∈ {5, 10, 100, 1000} members each. Adversarial examples are generated by attacking the ensemble directly until the generated image completely fools the ensemble. For HyperGAN, we attack the ensemble, but test with a new ensemble of equal size. For MNF we follow their experimental setup and do not form an ensemble. Rather we attack sampled networks and measure the entropy of their predictions.
For the purposes of detection, we compute the entropy within the predictive distribution of each of the ensemble members to score the example on the likelihood that it was drawn from the training distribution. Figure 5 shows that HyperGAN clearly identifies adversarial examples as being out-ofdistribution, performing better than MNF as well as standard ensembles. HyperGAN is especially suited to this task as adversarial examples are optimized against parameters - parameters which HyperGAN can change. We find that we can successfully detect over 97% of adversarial examples, with a low false positive rate for both attacks just by thresholding the entropy.

heading: 5 DISCUSSION AND FUTURE DIRECTIONS

text: We have proposed a generative model for parameter selection which performs strongly on detecting out-of-distribution samples as well as classification. Training a GAN to learn a probability distribution over parameters allows us to non-deterministically sample diverse, performant networks which we can use to form ensembles that can give good uncertainty estimates. Our method is ultimately scalable to any number of networks in the predicting ensemble, requiring just one forward pass to generate a new set of parameters and a low GPU memory footprint. We showed that we can generate models with significant variation over the learned distribution and thus provide uncertainty estimates on outlier data. There is still much room for exploration, we believe that HyperGAN may be useful in a variety of domains including meta learning and reinforcement learning.

heading: A APPENDIX

heading: A.1 GENERATED FILTER EXAMPLES

text: We show the first filter in 25 different networks generated by the HyperGAN to illustrate their difference in Fig. 7. It can be seen that qualitatively HyperGAN learns to generate classifiers with a variety of filters.

heading: A.2 OUTLIER EXAMPLES

text:
In Figure 7 we show images of examples which do not behave like most of their respective distribution. On top are MNIST images which HyperGAN networks predict to have high entropy. We can see that they are generally ambiguous and do not fit with the rest of the training data. The bottom
row shows notMNIST examples which score with low entropy according to HyperGAN. It can be seen that these examples look like they could come from the MNIST training distribution, making HyperGAN’s predictions reasonable

heading: A.3 DIVERSITY WITHOUT MIXER OR DISCRIMINATOR

text: We run experiments on both MNIST and CIFAR-10 where we remove both the mixer and the discriminator. Tables 3 and 4 show statistics of the networks generated by HyperGAN using only independent Gaussian samples to the generators. In this setting, HyperGAN learns to generate only a very small distribution of parameters.

heading: A.4 HYPERGAN NETWORK DETAILS

text:
In tables 5 and 6 we show how the latent points are transformed through the generators to become a full layer of parameters. For a MNIST based HyperGAN we generate layers from small latent points
of dimensionality 128. For CIFAR-10 based HyperGANs we use a larger dimensionality of 256 for the latent points.
Table 6: CIFAR-10 HyperGAN Target Size
Layer Latent Size Output Layer Size
Conv 1 256 x 1 16 x 3 x 3 x 3 Conv 2 256 x 1 32 x 16 x 3 x 3 Conv 3 256 x 1 32 x 64 x 3 x 3 Linear 1 256 x 1 256 x 128 Linear 2 256 x 1 128 x 10

heading: A.5 CONNECTIONS TO GANS AND WASSERSTEIN AUTO-ENCODERS

text:
Despite being largely applied to the same task of learning probability distributions, implicit generative models such as Generative Adversarial Networks (GAN) and Wasserstein Auto-encoders (WAE) are presented under different frameworks. GANs are a likelihood free density estimator, specifying only a generator, and do not have any notion of reconstruction. WAEs find an optimal transport map between target and model distributions by first encoding samples to a latent space and reconstructing the original data samples using a generator. Even with these differences, many combinations of these models have been proposed. Creating composite models and loss functions from the GAN and WAE building blocks is commonplace.
HyperGAN is also an implicit generative model, in that there is no explicit data distribution from which to sample from. HyperGAN learns to estimate from the data distribution not from real samples, but from the error of its estimates. We believe that HyperGAN is not just a combination of existing generative models. To show this we give an overview of both GANs and WAE, and contrast HyperGAN to them.

heading: GENERATIVE ADVERSARIAL NETWORKS

text:
GANs in the original formulation (Goodfellow et al., 2014) play a minimax game between two neural networks. A generator G : Z → X samples a prior distribution PZ on the latent space Z and attempts to produce outputs from the data distribution X . A discriminator D evaluates samples from X and from G(Z) and outputs a score between [0, 1]. This score given by the discriminator corresponds to how likely the input is to have come from X , according to D. This creates the following minimax game between the generator and the discriminator:
min G max D Ex∼X [logD(x)] + Ez∼PZ [log 1−D(G(z))] (7)
The goal of the generator is to learn to transform noise samples into samples which fool the discriminator, thus minimizing the loss on G. While the discriminator tries to maximize its ability to tell real images from fake samples from the generator.
In contrast, HyperGAN is not a likelihood free model like a GAN. While GANs do not specify an explicit evaluation criterion, HyperGAN minimizes the negative likelihood of the generated parameters with an explicit criterion: the loss on the target task. HyperGAN then has the benefit of a principled quantitative evaluation of its samples, and does not need to rely on metrics like the Inception (Salimans et al., 2016) or FID scores (Heusel et al., 2017) commonly used with GANs.

heading: WASSERSTEIN AUTO-ENCODERS

text:
Wasserstein Auto-encoders (WAE) (Tolstikhin et al., 2017) use a similar formulation under the optimal transport framework (Villani, 2008) to minimize the difference between generative model PG and the data distribution PX , where X ∈ X in this case are images. In the WAE, A latent space Z and prior distribution PZ on Z are defined as sample spaces. In general, there is an optimal transport map between PX and PG if the data distribution PX is well distributed according to a prior distribution (in this case PZ). The problem is that the data distribution PX is not a well defined distribution, and so computation of an OT mapping is intractable. To remedy this, an encoder Q : X → Z is used to project points from the data distribution to the latent space. WAE considers the marginal
distribution Z ∼ Q(Z|X). The conditional Q(Z|X) represents the aggregated posterior QZ : the space of Z which we can optimize over with an encoder. The authors argue that this allows WAE to not have to search over all possible maps PX → PG, but only those originating from the better behaved QZ . The encoder’s aggregated posterior QZ still needs to be distributed according to prior PZ , so a penalty is imposed onQZ to match the mean and covariance of PZ . In principle the penalty term DZ(PZ , QZ) could be any distance between probability distributions. WAE uses either MMD or a discriminator trained with an adversarial loss.
inf G inf Q(Z|X)∈Q EPXEQ(Z|X) [c(X,G(Z))] + λDZ(PZ , QZ) (8)
Where c is any measurable cost function measuring the error between the data X and reconstruction G(Z), and λ is a hyperparameter.
While HyperGAN and WAE use a similar architecture, HyperGAN is not an auto-encoder. As such, HyperGAN replaces the WAE reconstruction cost function c(X,G(Z)) with a term that does not depend on X . Instead, X is hidden, and HyperGAN minimizes the KL divergence between estimates θG ∈ ΘG and the true parameter distribution Θ. Because the Wasserstein distance does not reduce to the KL divergence, we can not exactly compute the optimal transport map:
inf G inf Q(Z|S)∈Q c[S,G(Z)] (9)
because we cannot compute the reconstruction cost c. To compute cwe need samples from the target distribution. Instead, we can only approximate the optimal map by minimizing
inf G inf Q(Z|S)∈Q
L(F(X;G(Z), Y )). (10)
Which nonetheless gives us well-behaved samples from a learned distribution of parameters.