summaries: This paper targets the very important problem of reward-hacking that occurs when the objectives optimized by intelligent agents are misaligned with respect to the tru objectives of the algorithm designer. The paper presents an empirical study across a range of different settings including a simple driving simulator, covid modeling, and a single atari game. The experiments show evidence of reward hacking as a function of modeling power of the agent and the size of the state-space. The paper concludes with some ideas and initial directions on how to potentially mitigate reward hacking.

summaries: This paper studies reward hacking, a common but understudied phenomenon, across a set of environments. Reward hacking emerges in several tasks, meaning that the resulting policy has a high proxy reward but a low true reward. A key finding is that reward hacking increases with agent capabilities so that increasing capability lowers the true reward. This holds across several ways of increasing capabilities (model size, training steps, action space, etc). The authors also find ‘phase transitions’ where a small increase in capability results in qualitatively new reward hacking behavior, a phenomenon that may require novel monitoring strategies. One such strategy is anomaly detection, for which the authors introduce a benchmark and baselines.

summaries: This paper provides a systematic study of “reward hacking” in the environments with the misspecified rewards. The authors conduct a set of experiments with 4 environments, several types of reward misspecification in each of them and several agents of different expressivity (model capacity). They notice that often the agents that are more capable end up obtaining high proxy reward, but low real reward. Besides, often the transition to the low real reward happens very quickly and authors call this phenomenon “phase transition”. Finally, they propose a baseline for anomaly detection to identify this phase transition.

summaries: This paper investigates the phenomenon of reward hacking as a function of agent capabilities. They introduce four diverse RL environments with nine misspecified rewards and demonstrate that more capable agents are better at exploiting the misspecification. They find instances of phase transitions where a small increase in agent capability produces a large change in behavior that sharply decreases the true reward.  To mitigate the reward hacking problem, they propose to set up an anomaly detection task, given a trusted model with moderate performance on the true reward, where the anomaly detector's task is to identify whether policies from a different model are satisfactory for the true reward. They provide several baseline anomaly detectors and show how they perform on different tasks.

abstractText: Reward hacking—where RL agents exploit gaps in misspecified reward functions—has been widely observed, but not yet systematically studied. To understand how reward hacking arises, we construct four RL environments with misspecified rewards. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. More capable agents often exploit reward misspecifications, achieving higher proxy reward and lower true reward than less capable agents. Moreover, we find instances of phase transitions: capability thresholds at which the agent’s behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.

authors:
Marcin Andrychowicz
Anton Raichuk
Piotr Stańczyk
Manu Orsini
Sertan Girgin
Raphael Marinier
Léonard Hussenot
Matthieu Geist
Olivier Pietquin
Marcin Michalski

title: What matters in on-policy reinforcement learning? A large-scale empirical study

venue: arXiv preprint arXiv:2006.05990,

authors:
Bowen Baker
Ingmar Kanitscheider
Todor Markov
Yi Wu
Glenn Powell
Bob McGrew
Igor Mordatch

title: Emergent tool use from multi-agent autocurricula

venue: In International Conference on Learning Representations,

authors:
Peter Boettke
Benjamin Powell

title: The political economy of the covid-19 pandemic

venue: Southern Economic Journal,

title: On the opportunities and risks of foundation models

venue: arXiv preprint arXiv:2108.07258,

authors:
Amy Bronstone
Claudia Graham

title: The potential cost implications of averting severe hypoglycemic events requiring hospitalization in high-risk adults with type 1 diabetes using real-time continuous glucose monitoring

venue: Journal of Diabetes Science and Technology,

authors:
Daniel Brown
Russell Coleman
Ravi Srinivasan
Scott Niekum

title: Safe imitation learning via fast Bayesian reward inference from preferences

venue: In Proceedings of the 37th International Conference on Machine Learning,

authors:
Paul F Christiano
Jan Leike
Tom Brown
Miljan Martic
Shane Legg
Dario Amodei

title: Deep reinforcement learning from human preferences

venue: In Advances in Neural Information Processing Systems,

authors:
Jesse Davis
Mark Goadrich

title: The relationship between precision-recall and roc curves

authors:
Jasha Droppo
Oguz Elibol

title: Scaling laws for acoustic models

venue: arXiv preprint arXiv:2106.09488,

authors:
Lasse Espeholt
Hubert Soyer
Remi Munos
Karen Simonyan
Volodymyr Mnih
Tom Ward
Yotam Doron
Vlad Firoiu
Tim Harley
Iain Robert Dunning
Shane Legg
Koray Kavukcuoglu

title: Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. 2018

authors:
Tom Everitt
Victoria Krakovna
Laurent Orseau
Shane Legg

title: Reinforcement learning with a corrupted reward channel

venue: In International Joint Conference on Artificial Intelligence,

authors:
Ian Fox
Joyce Lee
Rodica Pop-Busui
Jenna Wiens

title: Deep reinforcement learning for closedloop blood glucose control

venue: In Machine Learning for Healthcare Conference,

authors:
M. Fralick
A.S. Kesselheim

title: The U.S. Insulin Crisis - Rationing a Lifesaving Medication Discovered in the 1920s

venue: New England Journal of Medicine,

authors:
Ian J Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

venue: arXiv preprint arXiv:1412.6572,

authors:
Tuomas Haarnoja
Aurick Zhou
Pieter Abbeel
Sergey Levine

title: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

venue: In International conference on machine learning,

authors:
Dylan Hadfield-Menell
Smitha Milli
Pieter Abbeel
Stuart J Russell
Anca Dragan

title: Inverse reward design

authors:
Dan Hendrycks
Kevin Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks

venue: International Conference on Learning Representations,

authors:
Dan Hendrycks
Nicholas Carlini
John Schulman
Jacob Steinhardt

title: Unsolved problems in ml safety

venue: arXiv preprint arXiv:2109.13916,

authors:
Dan Hendrycks
Mantas Mazeika
Andy Zou
Sahil Patel
Christine Zhu
Jesus Navarro
Dawn Song
Bo Li
Jacob Steinhardt

title: What would Jiminy Cricket do? Towards agents that behave morally

authors:
Darby Herkert
Pavithra Vijayakumar
Jing Luo
Jeremy I. Schwartz
Tracy L. Rabin
Eunice DeFilippo
Kasia J. Lipska

title: Cost-related insulin underuse among patients with diabetes

venue: JAMA Internal Medicine,

authors:
Danny Hernandez
Jared Kaplan
Tom Henighan
Sam McCandlish

title: Scaling laws for transfer

venue: arXiv preprint arXiv:2102.01293,

authors:
Evan Hubinger
Chris van Merwijk
Vladimir Mikulik
Joar Skalse
Scott Garrabrant

title: Risks from learned optimization in advanced machine learning systems

authors:
Borja Ibarz
J. Leike
Tobias Pohlen
Geoffrey Irving
S. Legg
Dario Amodei

title: Reward learning from human preferences and demonstrations in Atari

authors:
Zaynah Javed
Daniel S Brown
Satvik Sharma
Jerry Zhu
Ashwin Balakrishna
Marek Petrik
Anca Dragan
Ken Goldberg

title: Policy gradient bayesian robust optimization for imitation learning

venue: In Proceedings of the 38th International Conference on Machine Learning,

authors:
Jared Kaplan
Sam McCandlish
Tom Henighan
Tom B Brown
Benjamin Chess
Rewon Child
Scott Gray
Alec Radford
Jeffrey Wu
Dario Amodei

title: Scaling laws for neural language models

authors:
W. Bradley Knox
Alessandro Allievi
Holger Banzhaf
Felix Schmitt
Peter Stone

title: Reward (Mis)design for Autonomous Driving

venue: arXiv e-prints

authors:
Jens Kober
J Andrew Bagnell
Jan Peters

title: Reinforcement learning in robotics: A survey

venue: The International Journal of Robotics Research,

authors:
Varun Kompella
Roberto Capobianco
Stacy Jong
Jonathan Browne
Spencer Fox
Lauren Meyers
Peter Wurman
Peter Stone

title: Reinforcement learning for optimization of covid-19 mitigation

authors:
BorIs. P. Kovatchev
Martin Straume
Daniel J. Cox
Leon.S Farhy

title: Risk analysis of blood glucose data:a quantitative approach to optimizing the control of insulin dependent diabetes

venue: Journal of Theoretical Medicine,

authors:
Heinrich Küttler
Nantas Nardelli
Thibaut Lavril
Marco Selvatici
Viswanath Sivakumar
Tim Rocktäschel
Edward Grefenstette

title: TorchBeast: A PyTorch Platform for Distributed RL

title: How much does insulin cost? Here’s how 23 brands compare

authors:
Jan Leike
Miljan Martic
Victoria Krakovna
Pedro A. Ortega
Tom Everitt
Andrew Lefrancq
Laurent Orseau
Shane Legg

title: AI safety gridworlds, 2017

authors:
Michael L. Littman
Ifeoma Ajunwa
Guy Berger
Craig Boutilier
Morgan Currie
Finale DoshiVelez
Gillian Hadfield
Michael C. Horowitz
Charles Isbell
Hiroaki Kitano
Karen Levy
Terah Lyons
Melanie Mitchell
Julie Shah
Steven Sloman
Shannon Vallor
Toby Walsh

title: Gathering strength, gathering storms: The one hundred year study on artificial intelligence (AI100) 2021 study panel report

authors:
Pablo Alvarez Lopez
Michael Behrisch
Laura Bieker-Walz
Jakob Erdmann
Yun-Pang Flötteröd
Robert Hilbrich
Leonhard Lücken
Johannes Rummel
Peter Wagner
Evamarie Wießner

title: Microscopic traffic simulation using SUMO

venue: In International Conference on Intelligent Transportation Systems,

authors:
Chiara Dalla Man
Francesco Micheletto
Dayu Lv
Marc Breton
Boris Kovatchev
Claudio Cobelli

title: The UVA/PADOVA type 1 diabetes simulator: New features

authors:
Romain Paulus
Caiming Xiong
Richard Socher

title: A deep reinforced model for abstractive summarization

authors:
Manoel Horta Ribeiro
Raphael Ottoni
Robert West
Virgı́lio A.F. Almeida
Wagner Meira

title: Auditing radicalization pathways on youtube

venue: In Conference on Fairness, Accountability, and Transparency,

title: Human Compatible: Artificial Intelligence and the Problem of Control

authors:
John Schulman
Filip Wolski
Prafulla Dhariwal
Alec Radford
Oleg Klimov

title: Proximal policy optimization algorithms

venue: arXiv preprint arXiv:1707.06347,

authors:
Nisan Stiennon
Long Ouyang
Jeff Wu
Daniel M Ziegler
Ryan Lowe
Chelsea Voss
Alec Radford
Dario Amodei
Paul Christiano

title: Learning to summarize from human feedback

title: Aligning ai optimization to community well-being

venue: International Journal of Community Well-Being,

authors:
Jihoon Tack
Sangwoo Mo
Jongheon Jeong
Jinwoo Shin

title: Csi: Novelty detection via contrastive learning on distributionally shifted instances

venue: Advances in Neural Information Processing Systems,

title: Quantilizers: A safer alternative to maximizers for limited optimization

venue: In AAAI Workshop: AI, Ethics, and Society,

authors:
Marin Toromanoff
Emilie Wirbel
Fabien Moutarde

title: Is deep reinforcement learning really superhuman on Atari? Leveling the playing field, 2019

authors:
Martin Treiber
Ansgar Hennecke
Dirk Helbing

title: Congested traffic states in empirical observations and microscopic simulations

venue: Physical review E,

authors:
Alexander Trott
Sunil Srinivasa
Douwe van der Wal
Sebastien Haneuse
Stephan Zheng

title: Building a Foundation for Data-Driven, Interpretable, and Robust Policy Design using the AI Economist

venue: arXiv preprint arXiv:2108.02904,

authors:
Eugene Vinitsky
Aboudy Kreidieh
Luc Le Flem
Nishant Kheterpal
Kathy Jang
Cathy Wu
Fangyu Wu
Richard Liaw
Eric Liang
Alexandre M. Bayen

title: Benchmarks for reinforcement learning in mixed-autonomy traffic

venue: In Conference on Robot Learning,

authors:
Cathy Wu
Abdul Rahman Kreidieh
Kanaad Parvate
Eugene Vinitsky
Alexandre M. Bayen

title: Flow: A modular learning framework for mixed autonomy traffic

venue: IEEE Transactions on Robotics,

title: Self-organizing systems: The emergence of order

venue: Springer Science & Business Media,

authors:
Chao Yu
Jiming Liu
Shamim Nemati

title: Reinforcement learning in healthcare: A survey

venue: arXiv preprint arXiv:1908.08796,

authors:
Simon Zhuang
Dylan Hadfield-Menell

title: Consequences of misaligned AI

heading: 1 INTRODUCTION

text:
As reinforcement learning agents are trained with better algorithms, more data, and larger policy models, they are at increased risk of overfitting their objectives (Russell, 2019). Reward hacking, or the gaming of misspecified reward functions by RL agents, has appeared in a variety of contexts, such as game playing (Ibarz et al., 2018), text summarization (Paulus et al., 2018), and autonomous driving (Knox et al., 2021). These examples show that better algorithms and models are not enough; for human-centered applications such as healthcare (Yu et al., 2019), economics (Trott et al., 2021) and robotics (Kober et al., 2013), RL algorithms must be safe and aligned with human objectives (Bommasani et al., 2021; Hubinger et al., 2019).
Reward misspecifications occur because real-world tasks have numerous, often conflicting desiderata. In practice, reward designers resort to optimizing a proxy reward that is either more readily measured or more easily optimized than the true reward. For example, consider a recommender system optimizing for users’ subjective well-being (SWB). Because SWB is difficult to measure, engineers rely on more tangible metrics such as click-through rates or watch-time. Optimizing for misspecified proxies led YouTube to overemphasize watch-time and harm user satisfaction (Stray, 2020), as well as to recommended extreme political content to users (Ribeiro et al., 2020).
Addressing reward hacking is a first step towards developing human-aligned RL agents and one goal of ML safety (Hendrycks et al., 2021a). However, there has been little systematic work investigating when or how it tends to occur, or how to detect it before it runs awry. To remedy this, we study the problem of reward hacking across four diverse environments: traffic control (Wu et al., 2021), COVID response (Kompella et al., 2020), blood glucose monitoring (Fox et al., 2020), and the Atari game Riverraid (Brockman et al., 2016). Within these environments, we construct nine misspecified proxy reward functions (Section 3).
Using our environments, we study how increasing optimization power affects reward hacking, by training RL agents with varying resources such as model size, training time, action space resolution, and observation space noise (Section 4). We find that more powerful agents often attain higher proxy reward but lower true reward, as illustrated in Figure 1. Since the trend in ML is to increase resources exponentially each year (Littman et al., 2021), this suggests that reward hacking will become more pronounced in the future in the absence of countermeasures.
More worryingly, we observe several instances of phase transitions. In a phase transition, the more capable model pursues a qualitatively different policy that sharply decreases the true reward. Figure 1 illustrates one example: An RL agent regulating traffic learns to stop any cars from merging onto the highway in order to maintain a high average velocity of the cars on the straightaway.
Since there is little prior warning of phase transitions, they pose a challenge to monitoring the safety of ML systems. Spurred by this challenge, we propose an anomaly detection task (Hendrycks & Gimpel, 2017; Tack et al., 2020): Can we detect when the true reward starts to drop, while maintaining a low false positive rate in benign cases? We instantiate our proposed task, POLYNOMALY, for the traffic and COVID environments (Section 5). Given a trusted policy with moderate performance, one must detect whether a given policy is aberrant. We provide several baseline anomaly detectors for this task and release our data at https://github.com/aypan17/ reward-misspecification.

heading: 2 RELATED WORK

text:
Previous works have focused on classifying different types of reward hacking and sometimes mitigating its effects. One popular setting is an agent on a grid-world with an erroneous sensor. HadfieldMenell et al. (2017) show and mitigate the reward hacking that arises due to an incorrect sensor reading at test time in a 10x10 navigation grid world. Leike et al. (2017) show examples of reward hacking in a 3x3 boat race and a 5x7 tomato watering grid world. Everitt et al. (2017) theoretically study and mitigate reward hacking caused by a faulty sensor.
Game-playing agents have also been found to hack their reward. Baker et al. (2020) exhibit reward hacking in a hide-and-seek environment comprising 3-6 agents, 3-9 movable boxes and a few ramps: without a penalty for leaving the play area, the hiding agents learn to endlessly run from the seeking agents. Toromanoff et al. (2019) briefly mention reward hacking in several Atari games (Elevator Action, Kangaroo, Bank Heist) where the agent loops in a sub-optimal trajectory that provides a repeated small reward.
Agents optimizing a learned reward can also demonstrate reward hacking. Ibarz et al. (2018) show an agent hacking a learned reward in Atari (Hero, Montezuma’s Revenge, and Private Eye), where optimizing a frozen reward predictor eventually achieves high predicted score and low actual score. Christiano et al. (2017) show an example of reward hacking in the Pong game where the agent learns to hit the ball back and forth instead of winning the point. Stiennon et al. (2020) show that a policy which over-optimizes the learnt reward model for text summarization produces lower quality summarizations when judged by humans.

heading: 3 EXPERIMENTAL SETUP: ENVIRONMENTS AND REWARD FUNCTIONS

text: In this section, we describe our four environments (Section 3.1) and taxonomize our nine corresponding misspecified reward functions (Section 3.2).

heading: 3.1 ENVIRONMENTS

text:
We chose a diverse set of environments and prioritized complexity of action space, observation space, and dynamics model. Our aim was to reflect real-world constraints in our environments, selecting ones with several desiderata that must be simultaneously balanced. Table 1 provides a summary.
Traffic Control. The traffic environment is an autonomous vehicle (AV) simulation that models vehicles driving on different highway networks. The vehicles are either controlled by a RL algorithm or pre-programmed via a human behavioral model. Our misspecifications are listed in Table 1.
We use the Flow traffic simulator, implemented by Wu et al. (2021) and Vinitsky et al. (2018), which extends the popular SUMO traffic simulator (Lopez et al., 2018). The simulator uses cars that drive like humans, following the Intelligent Driver Model (IDM) (Treiber et al., 2000), a widely-accepted approximation of human driving behavior. Simulated drivers attempt to travel as fast as possible while tending to decelerate whenever they are too close to the car immediately in front.
The RL policy has access to observations only from the AVs it controls. For each AV, the observation space consists of the car’s position, its velocity, and the position and velocity of the cars immediately in front of and behind it. The continuous control action is the acceleration applied to each AV. Figure 4 depicts the Traffic-Mer network, where cars from an on-ramp attempt to merge onto the straightaway. We also use the Traffic-Bot network, where cars (1-4 RL, 10-20 human) drive through a highway bottleneck where lanes decrease from four to two to one.
COVID Response. The COVID environment, developed by Kompella et al. (2020), simulates a population using the SEIR model of individual infection dynamics. The RL policymaker adjusts the severity of social distancing regulations while balancing economic health (better with lower regulations) and public health (better with higher regulations), similar in spirit to Trott et al. (2021). The population attributes (proportion of adults, number of hospitals) and infection dynamics (random testing rate, infection rate) are based on data from Austin, Texas.
Every day, the environment simulates the infection dynamics and reports testing results to the agent, but not the true infection numbers. The policy chooses one of three discrete actions: INCREASE, DECREASE, or MAINTAIN the current regulation stage, which directly affects the behavior of the population and indirectly affects the infection dynamics. There are five stages in total.
Atari Riverraid. The Atari Riverraid environment is run on OpenAI Gym (Brockman et al., 2016). The agent operates a plane which flies over a river and is rewarded by destroying enemies. The agent observes the raw pixel input of the environment. The agent can take one of eighteen discrete actions, corresponding to either movement or shooting within the environment.
Glucose Monitoring. The glucose environment, implemented in Fox et al. (2020), is a continuous control problem. It extends a FDA-approved simulator (Man et al., 2014) for blood glucose levels of a patient with Type 1 diabetes. The patient partakes in meals and wears a continuous glucose monitor (CGM), which gives noisy observations of the patient’s glucose levels. The RL agent administers insulin to maintain a healthy glucose level.
Every five minutes, the agent observes the patient’s glucose levels and decides how much insulin to administer. The observation space is the previous four hours of glucose levels and insulin dosages.

heading: 3.2 MISSPECIFICATIONS

text:
Using the above environments, we constructed nine instances of misspecified proxy rewards. To help interpret these proxies, we taxonomize them as instances of misweighting, incorrect ontology, or incorrect scope. We elaborate further on this taxonimization using the traffic example from Figure 1.
• Misweighting. Suppose that the true reward is a linear combination of commute time and acceleration (for reducing carbon emissions). Downweighting the acceleration term thus underpenalizes carbon emissions. In general, misweighting occurs when the proxy and true reward capture the same desiderata, but differ on their relative importance.
• Ontological. Congestion could be operationalized as either high average commute time or low average vehicle velocity. In general, ontological misspecification occurs when the proxy and true reward use different desiderata to capture the same concept.
• Scope. If monitoring velocity over all roads is too costly, a city might instead monitor them only over highways, thus pushing congestion to local streets. In general, scope misspecification occurs when the proxy measures desiderata over a restricted domain (e.g. time, space).
We include a summary of all nine tasks in Table 1 and provide full details in Appendix A. Table 1 also indicates whether each proxy leads to misalignment (i.e. to a policy with low true reward) and whether it leads to a phase transition (a sudden qualitative shift as model capacity increases). We investigate both of these in Section 4.
Evaluation protocol. For each environment and proxy-true reward pair, we train an agent using the proxy reward and evaluate performance according to the true reward. We use PPO (Schulman et al., 2017) to optimize policies for the traffic and COVID environments, SAC (Haarnoja et al., 2018) to optimize the policies for the glucose environment, and torchbeast (Küttler et al., 2019), a PyTorch implementation of IMPALA (Espeholt et al., 2018), to optimize the policies for the Atari environment. When available, we adopt the hyperparameters (except the learning rate and network size) given by the original codebase.

heading: 4 HOW AGENT OPTIMIZATION POWER DRIVES MISALIGNMENT

text:
To better understand reward hacking, we study how it emerges as agent optimization power increases. We define optimization power as the effective search space of policies the agent has access to, as implicitly determined by model size, training steps, action space, and observation space.
In Section 4.1, we consider the quantitative effect of optimization power for all nine environmentmisspecification pairs; we primarily do this by varying model size, but also use training steps, action space, and observation space as robustness checks. Overall, more capable agents tend to overfit the proxy reward and achieve a lower true reward. We also find evidence of phase transitions on four of the environment-misspecification pairs. For these phase transitions, there is a critical threshold at which the proxy reward rapidly increases and the true reward rapidly drops.
In Section 4.2, we further investigate these phase transitions by qualitatively studying the resulting policies. At the transition, we find that the quantitative drop in true reward corresponds to a
qualitative shift in policy behavior. Extrapolating visible trends is therefore insufficient to catch all instances of reward hacking, increasing the urgency of research in this area.
In Section 4.3, we assess the faithfulness of our proxies, showing that reward hacking occurs even though the true and proxy rewards are strongly positively correlated in most cases.

heading: 4.1 QUANTITATIVE EFFECTS VS. AGENT CAPABILITIES

text:
As a stand-in for increasing agent optimization power, we first vary the model capacity for a fixed environment and proxy reward. Specifically, we vary the width and depth of the actor and critic networks, changing the parameter count by two to four orders of magnitude depending on the environment. For a given policy, the actor and critic are always the same size.
Model Capacity. Our results are shown in Figure 2, with additional plots included in Appendix A. We plot both the proxy (blue) and true (green) reward vs. the number of parameters. As model size increases, the proxy reward increases but the true reward decreases. This suggests that reward designers will likely need to take greater care to specify reward functions accurately and is especially salient given the recent trends towards larger and larger models (Littman et al., 2021).
The drop in true reward is sometimes quite sudden. We call these sudden shifts phase transitions, and mark them with dashed red lines in Figure 2. These quantitative trends are reflected in the qualitative behavior of the policies (Section 4.2), which typically also shift at the phase transition.
Model capacity is only one proxy for agent capabilities, and larger models do not always lead to more capable agents (Andrychowicz et al., 2020). To check the robustness of our results, we consider several other measures of optimization: observation fidelity, number of training steps, and action space resolution.
Number of training steps. Assuming a reasonable RL algorithm and hyperparameters, agents which are trained for more steps have more optimization power. We vary training steps for an agent trained on the Atari environment. The true reward incentivizes staying alive for as many frames as possible while moving smoothly. The proxy reward misweights these considerations by underpenalizing the smoothness constraint. As shown in Figure 3a, optimizing the proxy reward for more steps harms the true reward, after an initial period where the rewards are positively correlated.
Action space resolution. Intuitively, an agent that can take more precise actions is more capable. For example, as technology improves, an RL car may make course corrections every millisecond instead of every second. We study action space resolution in the traffic environment by discretizing the output space of the RL agent. Specifically, under resolution level ε, we round the action a ∈ R output by the RL agent to the nearest multiple of ε and use that as our action. The larger the resolution level ε, the lower the action space resolution. Results are shown in Figure 3b for a fixed model size. Increasing the resolution causes the proxy reward to remain roughly constant while the true reward decreases.
Observation fidelity. Agents with access to better input sensors, like higher-resolution cameras, should make more informed decisions and thus have more optimization power. Concretely, we study this in the COVID environment, where we increase the random testing rate in the population. The proxy reward is a linear combination of the number of infections and severity of social distancing, while the true reward also factors in political cost. As shown in Figure 3c, as the testing rate increases, the model achieves similar proxy reward at the cost of a slightly lower true reward.

heading: 4.2 QUALITATIVE EFFECTS

text:
In the previous section, quantitative trends showed that increasing a model’s optimization power often hurts performance on the true reward. We shift our focus to understanding how this decrease happens. In particular, we typically observe a qualitative shift in behavior associated with each of the phase transitions, three of which we describe below.
Traffic Control. We focus on the Traffic-Mer environment from Figure 2a, where minimizing average commute time is replaced by maximizing average velocity. In this case, smaller policies learn to merge onto the straightaway by slightly slowing down the other vehicles (Figure 4a). On the other hand, larger policy models stop the AVs to prevent them from merging at all (Figure 4b). This increases the average velocity, because the vehicles on the straightaway (which greatly outnumber vehicles on the on-ramp) do not need to slow down for merging traffic. However, it significantly increases the average commute time, as the passengers in the AV remain stuck.
COVID Response. Suppose the RL agent optimizes solely for the public and economic health of a society, without factoring politics into its decision-making. This behavior is shown in Figure 5. The larger model chooses to increase the severity of social distancing restrictions earlier than the smaller model. As a result, larger models are able to maintain low average levels of both ICU usage (a proxy for public health) and social distancing restrictions (a proxy for economic health). These
preemptive regulations may however be politically costly, as enforcing restrictions without clear signs of infection may foment public unrest (Boettke & Powell, 2021).
Atari Riverraid. We create an ontological misspecification by rewarding the plane for staying alive as long as possible while shooting as little as possible: a “pacifist run”. We then measure the game score as the true reward. We find that agents with more parameters typically maneuver more adeptly. Such agents shoot less frequently, but survive for much longer, acquiring points (true reward) due to passing checkpoints. In this case, therefore, the proxy and true rewards are wellaligned so that reward hacking does not emerge as capabilities increase.
We did, however, find that some of the agents exploited a bug in the simulator that halts the plane at the beginning of the level. The simulator advances but the plane itself does not move, thereby achieving high pacifist reward.
Glucose Monitoring. Consider an RL agent that optimizes solely for a patient’s health, without considering the economic costs of its treatment plans. In this case, the proxy reward is based off of a glycemic risk measure, which reflects the likelihood that a patient will suffer an acute hypoglycemic episode, developed by the medical community (Kovatchev et al., 2000).
However, a less economically-privileged patient may opt for the treatment plan with the least expected cost (Herkert et al., 2019; Fralick & Kesselheim, 2019), not the one with the least amount of risk. From this patient’s perspective, the true reward is the expected cost of the treatment plan, which includes the expected cost of hospital visits and the cost of administering the insulin.
Although larger model treatments reduce hypoglycemic risk more smaller model treatments, they administer more insulin. Based on the average cost of an ER visit for a hypogylcemic episode ($1350 from Bronstone & Graham (2016)) and the average cost of a unit of insulin ($0.32 from Lee (2020)), we find that it is actually more expensive to pursue the larger model’s treatment.

heading: 4.3 QUANTITATIVE EFFECTS VS PROXY-TRUE REWARD CORRELATION

text:
We saw in Sections 4.1 and 4.2 that agents often pursue proxy rewards at the cost of the true reward. Perhaps this only occurs because the proxy is greatly misspecified, i.e., the proxy and true reward are weakly or negatively correlated. If this were the case, then reward hacking may pose less of a threat. To investigate this intuition, we plot the correlation between the proxy and true rewards.
The correlation is determined by the state distribution of a given policy, so we consider two types of state distributions. Specifically, for a given model size, we obtain two checkpoints: one that achieves the highest proxy reward during training and one from early in training (less than 1% of training complete). We call the former the “trained checkpoint” and the latter the “early checkpoint”.
For a given model checkpoint, we calculate the Pearson correlation ρ between the proxy reward P and true reward T using 30 trajectory rollouts. Reward hacking occurs even though there is significant positive correlation between the true and proxy rewards (see Figure 6). The correlation is lower for the trained model than for the early model, but still high. Further figures are shown in Appendix A.2. Among the four environments tested, only the Traffic-Mer environment with ontological misspecification had negative Pearson correlation.

heading: 5 POLYNOMALY: MITIGATING REWARD MISSPECIFICATION

text:
In Section 4, we saw that reward hacking often leads to phase transitions in agent behaviour. Furthermore, in applications like traffic control or COVID response, the true reward may be observed only sporadically or not at all. Blindly optimizing the proxy in these cases can lead to catastrophic failure (Zhuang & Hadfield-Menell, 2020; Taylor, 2016).
This raises an important question: Without the true reward signal, how can we mitigate misalignment? We operationalize this as an anomaly detection task: the detector should flag instances of misalignment, thus preventing catastrophic rollouts. To aid the detector, we provide it with a trusted policy: one verified by humans to have acceptable (but not maximal) reward. Our resulting benchmark, POLYNOMALY, is described below.

heading: 5.1 PROBLEM SETUP

text:
We train a collection of policies by varying model size on the traffic and COVID environments. For each policy, we estimate the policy’s true reward by averaging over 5 to 32 rollouts. One author labeled each policy as acceptable, problematic, or ambiguous based on its true reward score relative to that of other policies. We include only policies that received a non-ambiguous label.
For both environments, we provide a small-to-medium sized model as the trusted policy model, as Section 4.1 empirically illustrates that smaller models achieve reasonable true reward without exhibiting reward hacking. Given the trusted model and a collection of policies, the anomaly detector’s task is to predict the binary label of “acceptable” or “problematic” for each policy.
Table 3 in Appendix B.1 summarizes our benchmark. The trusted policy size is a list of the hidden unit widths of the trusted policy network (not including feature mappings).

heading: 5.2 EVALUATION

text:
We propose two evaluation metrics for measuring the performance of our anomaly detectors.
• Area Under the Receiver Operating Characteristic (AUROC). The AUROC measures the probability that a detector will assign a random anomaly a higher score than a random non-anomalous policy (Davis & Goadrich, 2006). Higher AUROCs indicate stronger detectors.
• Max F-1 score. The F-1 score is the harmonic mean of the precision and the recall, so detectors with a high F-1 score have both low false positives and high true negatives. We calculate the max F-1 score by taking the maximum F-1 score over all possible thresholds for the detector.

heading: 5.3 BASELINES

text:
In addition to the benchmark datasets described above, we provide baseline anomaly detectors based on estimating distances between policies. We estimate the distance between the trusted policy and the unknown policy based on either the Jensen-Shannon divergence (JSD) or the Hellinger distance. Specifically, we use rollouts to generate empirical action distributions. We compute the distance between these action distributions at each step of the rollout, then aggregate across steps by taking either the mean or the range. For full details, see Appendix B.2. Table 2 reports the AUROC and F-1 scores of several such detectors. We provide full ROC curves in Appendix B.2.
We observe that different detectors are better for different tasks, suggesting that future detectors could do better than any of our baselines. Our benchmark and baseline provides a starting point for further research on mitigating reward hacking.

heading: 6 DISCUSSION

text:
In this work, we designed a diverse set of environments and proxy rewards, uncovered several instances of phase transitions, and proposed an anomaly detection task to help mitigate these transitions. Our results raise two questions: How can we not only detect phase transitions, but prevent them in the first place? And how should phase transitions shape our approach to safe ML?
On preventing phase transitions, anomaly detection already offers one path forward. Once we can detect anomalies, we can potentially prevent them, by using the detector to purge the unwanted behavior (e.g. by including it in the training objective). Similar policy shaping has recently been used to make RL agents more ethical (Hendrycks et al., 2021b). However, since the anomaly detectors will be optimized against by the RL policy, they need to be adversarially robust (Goodfellow et al., 2014). This motivates further work on adversarial robustness and adversarial anomaly detection. Another possible direction is optimizing policies against a distribution of rewards (Brown et al., 2020; Javed et al., 2021), which may prevent over-fitting to a given set of metrics.
Regarding safe ML, several recent papers propose extrapolating empirical trends to forecast future ML capabilities (Kaplan et al., 2020; Hernandez et al., 2021; Droppo & Elibol, 2021), partly to avoid unforeseen consequences from ML. While we support this work, our results show that trend extrapolation alone is not enough to ensure the safety of ML systems. To complement trend extrapolation, we need better interpretability methods to identify emergent model behaviors early on, before they dominate performance (Olah et al., 2018). ML researchers should also familiarize themselves with emergent behavior in self-organizing systems (Yates, 2012), which often exhibit similar phase transitions (Anderson, 1972). Indeed, the ubiquity of phase transitions throughout science suggests that ML researchers should continue to expect surprises–and should therefore prepare for them.

heading: ACKNOWLEDGEMENTS

text: We are thankful to Dan Hendrycks and Adam Gleave for helpful discussions about experiments and to Cassidy Laidlaw and Dan Hendrycks for providing valuable feedback on the writing. KB was supported by a JP Morgan AI Fellowship. JS was supported by NSF Award 2031985 and by Open Philanthropy.

heading: A MAPPING THE EFFECTS OF REWARD MISSPECIFICATION

text:
A.1 EFFECT OF MODEL SIZE
We plot the proxy and true reward vs. model size in Figure 7, following the experiment described in Section 4.1.
A.2 CORRELATION BETWEEN PROXY AND TRUE REWARDS
We plot the correlation between proxy and true rewards, following the experiment described in Section 4.3. Interestingly, we see that reward hacking still occurs when there is positive correlation between the true and proxy rewards, e.g., in Figures 8a/8b. Unsurprisingly, proxy-true pairs which are highly correlated, e.g., Figure 8c/8d do not exhibit reward hacking. Finally, proxy-true pairs which are negatively correlated, e.g., Figure 8e/8f exhibit the most reward hacking.

heading: B POLYNOMALY

text:
B.1 BENCHMARK STATISTICS
See Table 3 for Polynomaly’s statistics.
B.2 RECEIVER OPERATING CHARACTERISTIC CURVES
We plot the ROC curves for the detectors described in Section 5.3. Our detectors are calculated as follows.
Let P and Q represent two probability distributions with M = 12 (P +Q). Then the Jensen-Shannon divergence and the Hellinger distance between them is given by
JSD(P ||Q) := 1 2 KL(P ||M) + 1 2 KL(Q||M)
Hellinger(P,Q) := 1
2
∫ (√ dP − √ dQ )2 .
(1)
Our proposed detectors estimate the distance D(πtrusted, πunknown) between the trusted policy πtrusted and unknown policy πunknown as follows: We generate r rollouts of πunknown, where r = 5 in the traffic environment and r = 32 in the COVID environment. Every s steps of a rollout, where s = 10 in the traffic environment and s = 1 in the COVID environment, we set P to be the action distribution of πunknown given the unknown agent’s state at that timestep in the rollout and Q to be the action distribution of πtrusted given the unknown agent’s state at that timestep in the rollout. Intuitively, if P and Q are far apart, then the trusted agent would have performed a different action than the unknown agent at that given timestep, indicating a possible case of reward hacking. We then compute either JSD(P∥Q) or Hellinger(P,Q) following Equation (1). These distances are collected every s steps over the entire rollout, and we calculate metrics on these distances (range, mean, etc.) to assign an anomaly score to the untrusted policy.

title: THE EFFECTS OF REWARD MISSPECIFICATION: MAPPING AND MITIGATING MISALIGNED MODELS


summaries: The paper proposes a novel f-divergence Thermodynamic Variational Objective (f-TVO) framework for VI, that extends the TVO towards, a more general, family of f-divergences. The authors propose to use a $\chi$-deformed exponential distribution, which casts the f-TVO objective as integral along the $\chi$-path between p(x,z) and q(z|x) (rather than the geometric path in TVOs) under $\chi$-geometry. The authors propose different variants of f-TVO, that vary between the type of the f-divergance used, as well as how this integral is approximated (K-partitioned ) left-Riemann sum (related to ELBO), right-Riemann sum (related to EUBO) or a 'zig-zag' that alternates between the two. Besides theoretical justifications, results from two sets of experiments show that, in general, the proposed f-TVO perform comparable to or slightly better than the f-VI counterparts, but without clear conclusion wrt the choice of the f-divergence.

summaries: The paper presents a new bound as the objective for variational inference. The bound combines the recent progress of thermodynamic variational object which gives a tighter bound than the conventional ELBO, and the f-divergence which induces more possible distribution metrics. Experiments show its better performance on some Bayesian inference tasks.

summaries: This paper proposed an $f$-divergence TVO, which includes some existing works e.g., RVB, CUBO, ELBO into a unified framework. This paper's main idea is to transform $f$-divergence into a generalized $\chi$-exponenetial family and integral TVO along the $\chi$-path. The paper provides some theoretical analysis and the optimization methods of the suggested framework and supports the proposed $f$-TVO with numerical results.

summaries: This paper proposed new variational inference that combines f-divergence variational inference and the thermodynamic variational objective. The authors introduced several new concepts of exponential families to extend TVO. Finally, the authors provided the estimator of the gradient of the objective function based on the reparametrization trick.

abstractText: In this paper, we propose a f -divergence Thermodynamic Variational Objective (f -TVO). f -TVO generalizes the Thermodynamic Variational Objective (TVO) by replacing Kullback–Leibler (KL) divergence with arbitary differeitiable f divergence. In particular, f -TVO approximates dual function of model evidence f∗(p(x)) rather than the log model evidence log p(x) in TVO. f -TVO is derived from a deformed χ-geometry perspective. By defining χ-exponential family exponential, we are able to integral f -TVO along the χ-path, which is the deformed geodesic between variational posterior distribution and true posterior distribution. Optimizing scheme of f -TVO includes reparameterization trick and Monte Carlo approximation. Experiments on VAE and Bayesian neural network show that the proposed f -TVO performs better than cooresponding baseline f -divergence variational inference.

title: Information geometry and its applications, volume 194

authors:
David M Blei
Alp Kucukelbir
Jon D McAuliffe

title: Variational inference: A review for statisticians

venue: Journal of the American statistical Association,

authors:
Rob Brekelmans
Vaden Masrani
Frank Wood
Greg Ver Steeg
Aram Galstyan

title: All in the exponential family: Bregman duality in thermodynamic variational inference

venue: In International Conference on Machine Learning,

authors:
Yuri Burda
Roger B Grosse
Ruslan Salakhutdinov

title: Importance weighted autoencoders

venue: In ICLR (Poster),

authors:
Liqun Chen
Chenyang Tao
Ruiyi Zhang
Ricardo Henao
Lawrence Carin Duke

title: Variational inference and model selection with generalized evidence bounds

venue: In International conference on machine learning,

title: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Adji B Dieng
Dustin Tran
Rajesh Ranganath
John Paisley
David M Blei

title: Variational inference via x upper bound minimization

venue: In Proceedings of the 31st International Conference on Neural Information Processing Systems,

authors:
Huiping Duan
Linxiao Yang
Jun Fang
Hongbin Li

title: Fast inverse-free sparse bayesian learning via relaxed evidence lower bound maximization

venue: IEEE Signal Processing Letters,

authors:
Matthew D Hoffman
Matthew J Johnson

title: Elbo surgery: yet another way to carve up the variational evidence lower bound

venue: In Workshop in Advances in Approximate Bayesian Inference, NIPS,

authors:
Matthew D Hoffman
David M Blei
Chong Wang
John Paisley

title: Stochastic variational inference

venue: Journal of Machine Learning Research,

authors:
Alp Kucukelbir
Rajesh Ranganath
Andrew Gelman
David M Blei

title: Automatic variational inference in stan

venue: arXiv preprint arXiv:1506.03431,

authors:
Chris J Maddison
Dieterich Lawson
George Tucker
Nicolas Heess
Mohammad Norouzi
Andriy Mnih
Arnaud Doucet
Yee Whye Teh

title: Filtering variational objectives

venue: arXiv preprint arXiv:1705.09279,

authors:
Vaden W Masrani
Tuan Anh Le
Frank Wood

title: The thermodynamic variational objective

authors:
Andriy Mnih
Karol Gregor

title: Neural variational inference and learning in belief networks

authors:
Andriy Mnih
Danilo Rezende

title: Variational inference for monte carlo objectives

title: Generalised thermostatistics

authors:
John Paisley
David Blei
Michael Jordan

title: Variational bayesian inference with stochastic search

venue: arXiv preprint arXiv:1206.6430,

authors:
Rajesh Ranganath
Sean Gerrish
David Blei

title: Black box variational inference

venue: In Artificial intelligence and statistics,

authors:
Tim Salimans
David A Knowles

title: Fixed-form variational posterior approximation through stochastic linear regression

venue: Bayesian Analysis,

authors:
George Tucker
Andriy Mnih
Chris J Maddison
Dieterich Lawson
Jascha Sohl-Dickstein

title: Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models

venue: arXiv preprint arXiv:1703.07370,

authors:
Neng Wan
Dapeng Li
Naira Hovakimyan

title: f-divergence variational inference

title: Understanding the variational lower bound, 2017

authors:
Cheng Zhang
Judith Bütepage
Hedvig Kjellström
Stephan Mandt

title: Advances in variational inference

venue: IEEE transactions on pattern analysis and machine intelligence,

text: In this paper, we propose a f -divergence Thermodynamic Variational Objective (f -TVO). f -TVO generalizes the Thermodynamic Variational Objective (TVO) by replacing Kullback–Leibler (KL) divergence with arbitary differeitiable f - divergence. In particular, f -TVO approximates dual function of model evidence f∗(p(x)) rather than the log model evidence log p(x) in TVO. f -TVO is derived from a deformed χ-geometry perspective. By defining χ-exponential family exponential, we are able to integral f -TVO along the χ-path, which is the deformed geodesic between variational posterior distribution and true posterior distribution. Optimizing scheme of f -TVO includes reparameterization trick and Monte Carlo approximation. Experiments on VAE and Bayesian neural network show that the proposed f -TVO performs better than cooresponding baseline f -divergence variational inference.

heading: 1 INTRODUCTION

text:
Variational inference (VI) is a core technique in probabilistic machine learning (Mnih & Gregor, 2014; Mnih & Rezende, 2016; Tucker et al., 2017; Maddison et al., 2017; Paisley et al., 2012; Salimans et al., 2013; Ranganath et al., 2014; Kucukelbir et al., 2015). Normal Variational inference tries to perform a tractable posterior distribution approximation for log model evidence estimation log p(x). Commonly, this is done by introducing a divergence D[q(z|x)||p(z|x)] between variational distribution q(z|x) and true posterior p(z|x) as a regularization term. Then the evidence lower bound (ELBO): log p(x)−D[q(z|x)||p(z|x)] is used as a surrogate optimization objective for maximum likelihood parameter estimation (Hoffman & Johnson, 2016; Duan et al., 2017; Yang, 2017).
Recently, there are serveral directions that generalize normal variational inference. Interested readers can refer to VI’s background and progression in (Blei et al., 2017; Zhang et al., 2018). One direction is to explore tighter variational bounds (Burda et al., 2016; Chen et al., 2018; Masrani et al., 2019; Brekelmans et al., 2020). Among these efforts, the recent Thermodynamic Variational Objective (TVO) approach connects variational inference with thermodynamic integration. In paricular, TVO interprets the log model evidence estimation as 1D integral along the geodesic path between joint distribution of data and hidden variable p(x, z) and variational hidden variable distribution q(z|x). By properly choosing intergral intervals of Riemannian sum approximations, the TVO approximates the log likelihood with a series of upper and lower bounds, which are tighter surrogates than the normal variational inference objectives for model parameter likelihood maximization (Masrani et al., 2019).
Another direction is trying to extend the normal VI framework by replacing KL divergence with other statistical divergences. Among these efforts, Rényi’s α-divergence and χ-divergence can be regarded as the root divergences, which makes many existing divergence as special cases including KL-divergence. Rényi’s α-VI (Li & Turner, 2016) and χ-VI (Dieng et al., 2017) are stochastic VI that generalized from stochastic VI (Hoffman et al., 2013) and black-box VI (Ranganath et al., 2014), and outperform the classical KL-VI in Bayesian regressions and image reconstruction on some benchmarks. More generally, recent work f -divergence variational inference (f -VI) (Wan et al., 2020) gives a general framework that includes all divergences from f -divergence family. The variational objective of f -divergence VI can be regarded as the surrogate bound of dual function of model evidence f∗(p(x)).
In this paper, we combine the above two directions together by proposing f -TVO that generalizes the original TVO to arbitary f -divergence in a unified framework. f -TVO approximates dual function of model evidence f∗(p(x)) rather than the log model evidence log p(x) in TVO. The estimation of f -TVO is established from a deformed χ-geometry prospective. By defining χ-exponential family distribution, the computation of f -TVO is integrated over the χ-path, which is the geodesic between p(x, z) and q(z|x) under χ-geometry. The proposed f -TVO is a tighter bound of dual function of model evidence f∗(p(x)) rather than the normal variational lower bound in TVO.
Our contributions include the following: – We enrich the TVO by proposing a unified f-TVO framework that compatible with an amount
of existing divergence, such as KL, α divergence, χ divergence; – We derive a unified f-TVO bound equiped with the upper/lower bound criteria that improves
many existing bounds, such as ELBO, CUBO, RVB; – Experiments show that our proposed f-TVO is a better optimization surrogate than corre-
sponding f-VI. The rest of the paper is organized as follows: Sec. 2 reviews preliminary knowledges for f -divergence and Thermodynamic Variational Objective, Sec. 3 presents our f -TVO, Sec. 4 extends our approach to more variants of f -TVO, and Sec. 5 presents our experiment results.

heading: 2 PRELIMINARY

text:
In this section, we will review the preliminary knowledge for f -divergence and Thermodynamic Variational Objective.
2.1 f -DIVERGENCE
f -divergence is a measures of difference between two probability distributions. It can be defined as follows (Sason & Verdú, 2016).
Definition 1. Given a convex function f with f(1) = 0, then f -divergence from probability distribution q(z) to p(z) is defined as:
Df (p||q) = ∫ f( q(z)
p(z) )p(z)dz = Ep[f(
q(z) p(z) )] (1)
Many existing well-know divergence can be regarded as special cases of f -divergence by properly choose the generation function f . Table 1 shows the relationship between the sample divergences and their corresponding generation function f .
Wan et al.(Wan et al., 2020) proposed a f -divergence variational inference (f -VI) that generalizes normal variational inference to all f-divergences in a unified framework. f -VI framework is primarily based on reverse f -divergence (Wan et al., 2020). One can connect forward f -divergence Df (p||q) with the reverse f -divergence Df (q||p) via dual function f∗ as follows.
Definition 2. Given a function f, its dual funtion1 f∗ is defined as:
f∗(t) = t · f(1/t) (2)
The dual function f∗ has the following properties: (1)(f∗)∗ = f , (2)f∗ is convex iff f is convex, (3)f∗(1) = 0 iff f(1) = 0. f -divergence in equation 1 can be reformulated with the dual function f∗:
Df (p||q) = ∫ f( q(z)
p(z) )p(z)dz =
∫ f∗( p(z)
q(z) )q(z)dz = Df∗(q||p) (3)
As shown in the theorem below, f -variational bound is an approximation of the dual function f∗ of the model evidence p(x): Theorem 1. The dual function of model evidence f∗(p(x)) is bounded by the f -variational bound Lf (q, x)
Lf (q, x) = Eq(z|x)[f∗( p(z, x)
q(z|x) )] ≥ f∗(p(x)) (4)
where equality is obtained when q(z|x) = p(z|x).
The proof of the theorem is straightforward as f∗ is convex (Wan et al., 2020).
Many well-known variation bounds can be represented in the f -VI framework. For example, f -VI recovers to ELBO when f = t log t, and f -VI recovers to CUBOn when f = t1−n − t. More details are shown in the Table 1.

heading: 2.2 THERMODYNAMIC VARIATIONAL OBJECTIVE

text:
The evidence lower bound (ELBO) is a lower bound of the log evidence of a generative model p(x, z). It can be formulated as the difference between log evidence of model evidence log p(x) and KL divergence of variational posterior distribution q(z|x) and true posterior distribution p(z|x):
ELBO = log p(x)−DKL(q(z|x)||p(z|x)) (5)
The Thermodynamic Variational Objective (TVO) gives a tighter bound of log p(x) than ELBO. As shown in Fig. 1, TVO is a Riemannian sum of integral along the geometric path between the model and inference network:
1
K
( ELBO + K−1∑ k=1 Eπβk [log p(x, z) q(z|x) ] )
︸ ︷︷ ︸ TVO
≤ ∫ 1 0 Eπβ [log p(x, z) q(z|x) ]dβ = log p(x) (6)
1Note:dual function here is different from Fenchel conjugate function: fc(y) = supx∈dom f (y >x− f(x)).
where β = [β0, β1, · · · , βK ] is a partition between [0,1], where β0 = 0, and βK = 1, πβ is the normalized geometric combination of p(x, z) and q(z|x), i.e. π̃β = q(z|x)1−βp(x, z)β and πβ = π̃β/Zβ , where Zβ is a normalization constant for each β. One can verify that πβ is a member of exponential family distribution, no matter what q(z|x) and p(x, z) are:
πβ(z|x) = π0(z|x) exp{β · T (x, z)− φ(x;β)} (7)
where T (x, z) = log p(x,z)q(z|x) , φ(x;β) is the log partition function that normalize over z.
3 χ-THERMODYNAMIC VARIATIONAL OBJECTIVE
In this section, we establish our theory for f -Thermodynamic Variational Objective (f -TVO). We first connect f -divergence with χ-geometry. Then, we present f -TVO as an integration along the χ-path. Lastly, we specify the computation process of f -TVO and its gradients.
3.1 χ-LOGARITHM AND χ-EXPONENTIAL
We first define a generalized χ-logarithm function for a positive non-increasing function χ:
logχ(u) = ∫ u 1 χ(v)dv (8)
which is different from the definition in (Naudts, 2011; Amari, 2016). Notice that equation 8 gives normal log function log u when χ(v) = 1v . The inverse of the χ-logarithm is called χ-exponential:
expχ(u) = 1 + ∫ u 0 ψ(v)dv (9)
where ψ is an auxiliary function. One can determine ψ for each logχ(u), e.g. ψ(v) = exp(v) when logχ(u) = log u. χ-Logarithm and χ-Exponential have the following properties:
(1) d
du expχ(u) = ψ(u), (2) u = expχ(logχ(u)), (3)
1
χ(u) = ψ(logχ(u)).
For any divergence with convex diffrentiable generation function f with f(0) = 1, there exists a continuous positive non-increasing function χ that connects the dual function f∗:
f∗(u) = − logχ(u) = − ∫ u 1 χ(v)dv (10)
then f -divergence can be represented as Df (q||p) = Df∗(p||q) = − ∫ p(z) · logχ( q(z)
p(z) )dz (11)

heading: 3.2 THERMODYNAMIC INTEGRATION

text:
Using the χ-logarithm function and χ-exponential function defined in equation 8 and equation 9, we can define a exponential family distribution under χ-geometry:
Definition 3. χ-Exponential Family Distribution:
logχ p(x, θ)
c(x) = θ · x− Φ(θ) (12)
or equivalently:
p(x, θ) = c(x) expχ{θ · x− Φ(θ)} (13)
where x is random variable, θ is parameter, Φ(θ) is the partition function, c(x) is the normalization term.
Given two unnormalized distribution π̃0(z) and π̃1(z), we define an unnormalized distribution that is the log χ interpolation between π̃0(z) and π̃1(z):
logχ(π̃β(z)) = (1− β) logχ(π̃0(z)) + β logχ(π̃1(z)) (14) Its normalized distribution is defined as:
πβ(z) = π̃β(z)
Zβ (15) where Zβ = ∫ π̃β(z)dz. It’s easy to show that: π̃β(z) is on the log χ geodesic between π̃0(z) and π̃1(z) with χ-Exponential Family Distribution as: Theorem 2. πβ is also an χ-exponential family distribution, no matter π̃0(z) and π̃1(z).
Proof.
π̃β(z) = expχ((1− β) logχ π̃0(z) + β logχ π̃1(z)) = expχ(logχ π̃0(z) + β(logχ π̃1(z)− logχ π̃0(z))) (16)
By defining the potential energy function: Uβ(z) = logχ π̃β(z), then we can estimate the logχ of the ratio of the normalizing constants via thermodynamic integration:
logχ Z1 − logχ Z0 = ∫ 1 0 ∫ χ(Zβ) χ(π̃β(z)) (logχ π̃0(z)− logχ π̃1(z))dzdβ (17)
Notice that variational inference can be connected with thermodynamic integration if we set: π̃0(z) = q(z|x), Z0 = ∫ q(z|x)dz = 1
π̃1(z) = p(x, z), Z1 =
∫ p(x, z)dz = p(x) (18)
then the logχ of model evidence is the thermodynamic integration (TI):
logχ p(x) = ∫ 1 0 Sβdβ (19)
where Sβ = ∫ χ(Zβ) χ(π̃β(z)) (logχ q(z|x)− logχ p(x, z)))dz
Recall the relation between f -divergence and χ function in equation 10, then logχ p(x) = f ∗(p(x)) is dual function f of the model evidence, which corresponds to the f -VI in (Wan et al., 2020).
To approximate f -TVO defined in equation 17, we pick the a list of ordered partitions β = [β0, β1, · · · , βK ], where β0 ≤ β1 ≤ · · · ≤ βK , β0 = 0, βK = 1, then our f -TVO is the left Riemannian approximation of TI:
f -TVOL(K) = K∑ k=1 ∆kSβk−1 , (20)
where ∆k = βk − βk−1.

heading: 3.3 COMPUTATION

text:
In this section, we show how to efficiently compute the f -TVO.
We first rewrite Sβ in equation 17 into
Sβ =
∫ χ(Zβ)
χ(π̃β(z)) (logχ π̃0(z)− logχ π̃1(z))dz
= Eq(z)[ χ(Zβ)
χ(π̃β(z)) (logχ π̃0(z)− logχ π̃1(z))/q(z|x)] (21)
We apply the reparameterization trick to estimate Sβ . We assume there is a mapping gφ( ) that satisfies: z = gφ( ). Then the expectation of arbitrary function F (z) over distribution q(z) can be computed as Eq(z)[F (z)] = E [F (gφ( ))]. In particular, we use the prevalent Gaussian reparameterization: z ∼ N (µ,Σ), i.e. z = µ + Σ 12 , ∼ N (0, I). Then Sβ can be reformulated as:
Sβ = E [ χ(Zβ)
p( )χ(π̃β(gφ( ))) (logχ p( )− logχ p(x, gφ( )))] (22)
where we define q(z|x) = p( ) and p(x, z) = p(x, gφ( )) in equation 19. Empirically, we can compute Sβ as:
Ŝβ = 1
N N∑ n=1 [wn(logχ p( n)− logχ p(x, gφ( n)))] (23)
where wn = χ( ∑N n=1 π̃β(gφ( n)))/p( n)χ(π̃β(gφ( n))).
In practice, we split the computation of Ŝβ into two parts: Ŝβ = L−R:
L = 1
N N∑ n=1 [wn logχ p( n)]
R = 1
N N∑ n=1 [wn logχ p(x, gφ( n))] (24)
This is because we use log p(x, gφ( n)) and log p( n)) during our computation to avoid numerical overflow or underflow. All the intermediate sum operations are performed by logsumexp operation until we get logL and logR, then we use exponential operation to complete the computation.

heading: 4 GENERALIZING THERMODYNAMIC OBJECTIVES

text:
Recall f -TVO in equation 20 is left Riemannian approximation of Thermodynamic integration. Similarly, We can also have right Riemannian approximation of f -TI:
f -TVOR(K) = K∑ k=1 ∆kSβk (25)
Both left and right Riemannian approximation corresponds to existing variational bounds. For example, left single Riemannian approximation of the KL-TVO equals the ELBO, while the right single Riemannian approximation equals to the EUBO. Also, it is easy to verify if the dual function f∗ is increasing (decreasing), then left Riemannian approximation is an upper (lower) bound of Thermodynamic integration, while right Riemannian approximation is a lower (upper) bound of Thermodynamic integration.
In addition to the left or right Riemannian approximation of Thermodynamic integration, we can also define a “zig-zag” (z) approximation of Thermodynamic integration, which combines left and right Riemannian approximation. As shown in FIg. 2, the “zig-zag” Thermodynamic objective can be formulated as:
f -TVOZ = K∑ k=1 ∆k[1(k%2 = 1)Sβk−1 + 1(k%2 = 0)Sβk ] (26)
where 1(A) is indicator function, i.e. 1(A) = 1 when condition A is true, 1(A) = 0 when condition A is false.

heading: 5 EXPERIMENTS

text: In this section, we evaluate the effectiveness and the wide applicability of f -TVO on two tasks. f -TVO is first used as a surrogate optimization objective of f -VI in a Bayesian neural network for linear regression, then used in a VAE for image reconstruction and generation. For both tasks, we use Adam as our optimizer with recommended parameters in (Da, 2014). For all experiments, we use even partition of the temperature for Riemannian integration.

heading: 5.1 BAYESIAN VARIATIONAL AUTOENCODER

text:
f -TVO is used in Bayesian VAE for image reconstruction task on datasets including Caltech 101 Silhouettes (Cal), Frey Face (Fre), MNIST (MNI), and Omniglot (Omn). We have replaced the conventional ELBO loss function of VAE with the proposed f -TVOs that correspond to flexible f - divergences. We test and compare the f -TVO VAEs associated with three well-known f -divergences (KL-divergence, Renyi’s α-divergence with α = 3, and χ-divergence with n = 2). We also try a customized divergence which is defined by its dual function f∗c1(t) = log 2 t+ log t.
We use the left Riemannian approximation with 5 partitions, i.e. f -TVOL(5) in our setting. As shown in Table 2, the proposed f -TVOs outperforms the corresponding baselines in most cases.
Effects of numbers of partitions. Here, we study the effects of different numbers of partitions on Bayesian variational autoencoder in our experiments. We try partition numbers K = 1, 2, 5, 10 in the f -TVO, respectively. As shown in Fig. 3, we can see that even partition number is K = 2 can improve the baseline performance.
Effects of f -TVO variants. We also study the effects of f -TVO variants defined in Sec. 4. We try all 3 variants f -TVOL,f -TVOR and f -TVOZ on Bayesian variational autoencoder in our experiments. As shown in Fig. 4, three f -TVO variants achieve similar performance.

heading: 5.2 BAYESIAN NEURAL NETWORKS

text:
Our experiments of Bayesian neural network are performed on regression tasks. We generally follow the experiments settings as in (Wan et al., 2020). In particular, We use twelve datasets that are collected from the UCI dataset repository, where each dataset is randomly split into 90%/10% for training and testing. The model is a single-layer neural network with 50 hidden units (ReLUs). We use θ ∼ N (θ; 0, I) as a Gaussian prior of the network weights and Gaussian approximation to the true posterior.
In the experiments, we try different f -TVOs, including corresponds to KL TVO, α-divergence TVO, where α = 3, χ-square divergence TVO. Following (Wan et al., 2020), we also define a custom divergence, which is defined by its dual function fc2 = f̃∗(t) − f̃∗(1), where f̃∗(t) =
−1/6 · (log t + t0)3 − 1/2 · (log t + t0) − 1. We use the left Riemannian approximation with 5 partitions, i.e. f -TVOL(5) in our setting. As shown in Table 3, the proposed f -TVOs outperform the corresponding baselines.

heading: 6 CONCLUSION

text:
In this paper, we have proposed a general f -TVO framework that is a tighter evidence bounds than f -VI. f -TVO unifies all differential f -divergence into TVO from a χ-geometry perspective. By connecting f -divergence with χ-geometry, f -TVO is the Riemanian sum of thermodynamic
integration along the geodesic between q(z|x) and p(x, z) under χ-geometry. Empirical experiments of some instances of f -TVO on the popular benchmarks show the superior performance than stateof-the art results, which imply the flexibility and effectiveness of f -TVO. Future work on f -TVO may include more efficient f -TVO optimization methods, and more explorations on the properties of χ-geometry.