summaries: The paper demonstrates empirically that on a variety of architectures and datasets, the derivative of ReLU at 0 is evaluated a significant number of times during training in 16 and 32 bit precision. These events are frequent enough such that defining ReLU'(0) to be anything other than 0 (default value in common DNN frameworks) hurts generalization, and sometimes training as well.  The authors perform a variety of investigations around this effect, such as the influence of training set size, network width, depth, batch size, BatchNorm, Dropout.

summaries: This paper analyzes the impact of the choice of ReLU'[0] on the fidelity of neural networks. Even though one would expect that the value of ReLU'[0] should not have any noticeable impact, the authors notice the divergence in learned parametric values that depend on this choice. They also notice that this effect is more pronounced at lower precision (due to rounding effects that are dominant at fp16 and fp32 vs. fp64). They also study this effect for neural network sizes, topologies (including networks with and without batch normalization) and datasets.

summaries: The ReLU function is not differentiable at 0 which has a mathematical impact on ReLU neural networks. Nevertheless they are optimized using gradient based methods. The folk wisdom is that singular points only occur with zero probability and therefore the issue is irrelevant. This paper challenges this folk wisdom and identifies that the choice of the derivative at 0 does have a tangible impact on the optimization and classification performance.

summaries: This paper studies a long-neglected problem: the gradient of the non-smooth activation function in training a neural network. Specifically, the authors study the numerical influence of the ReLU derivative at zero on gradient descent. First, a theory is presented to illustrates the trivialness of ReLU'(0), then, the authors give counter-theory experiments to show that ReLU'(0) can affect training results.  In the last two sections, authors demonstrate that low-precision training, SGD optimizer, no batch norm training can magnify the impact of ReLU'(0).

abstractText: In theory, the choice of ReLU′(0) in [0, 1] for a neural network has a negligible influence both on backpropagation and training. Yet, in the real world, 32 bits default precision combined with the size of deep learning problems makes it a hyperparameter of training methods. We investigate the importance of the value of ReLU′(0) for several precision levels (16, 32, 64 bits), on various networks (fully connected, VGG, ResNet) and datasets (MNIST, CIFAR10, SVHN, ImageNet). We observe considerable variations of backpropagation outputs which occur around half of the time in 32 bits precision. The effect disappears with double precision, while it is systematic at 16 bits. For vanilla SGD training, the choice ReLU′(0) = 0 seems to be the most efficient. For our experiments on ImageNet the gain in test accuracy over ReLU′(0) = 1 was more than 10 points (two runs). We also evidence that reconditioning approaches as batch-norm or ADAM tend to buffer the influence of ReLU′(0)’s value. Overall, the message we convey is that algorithmic differentiation of nonsmooth problems potentially hides parameters that could be tuned advantageously.

authors:
M. Abadi
A. Agarwal
P. Barham
E. Brevdo
Z. Chen
C. Citro
G.S. Corrado
A. Davis
J. Dean
M. Devin
S. Ghemawat
I. Goodfellow
A. Harp
G. Irving
M. Isard
Y. Jia
R. Jozefowicz
L. Kaiser
M. Kudlur
J. Levenberg
D. Mané
R. Monga
S. Moore
D. Murray
C. Olah
M. Schuster
J. Shlens
B. Steiner
I. Sutskever
K. Talwar
P. Tucker
V. Vanhoucke
V. Vasudevan
F. Viégas
O. Vinyals
P. Warden
M. Wattenberg
M. Wicke
Y. Yu

title: and X

venue: Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems

authors:
M. Abadi
P. Barham
J. Chen
Z. Chen
A. Davis
J. Dean
M. Devin
S. Ghemawat
G. Irving
M. Isard

title: et al

venue: Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283

authors:
T. Akiba
S. Sano
T. Yanase
T. Ohta
M. Koyama

title: Optuna: A next-generation hyperparameter optimization framework

venue: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631

authors:
A.G. Baydin
B.A. Pearlmutter
A.A. Radul
J.M. Siskind

title: Automatic differentiation in machine learning: a survey

venue: Journal of machine learning research, 18

authors:
J. Berner
D. Elbrächter
P. Grohs
A. Jentzen

title: Towards a regularity theory for relu networks–chain rule and global error estimates

venue: 2019 13th International conference on Sampling Theory and Applications (SampTA), pages 1–5. IEEE

authors:
P. Bianchi
W. Hachem
S. Schechtman

title: Convergence of constant step stochastic gradient descent for non-smooth non-convex functions

venue: arXiv preprint arXiv:2005.08513

authors:
J. Bolte
E. Pauwels

title: Conservative set valued fields

venue: automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, pages 1–33

title: A mathematical model for automatic differentiation in machine learning

venue: Advances in Neural Information Processing Systems, volume 33

authors:
L. Bottou
F.E. Curtis
J. Nocedal

title: Optimization methods for large-scale machine learning

authors:
J. Bradbury
R. Frostig
P. Hawkins
M.J. Johnson
C. Leary
D. Maclaurin
G. Necula
A. Paszke
J. VanderPlas
S. Wanderman-Milne

title: and Q

venue: Zhang. JAX: composable transformations of Python+NumPy programs

authors:
M. Courbariaux
Y. Bengio
J.-P. David

title: Training deep neural networks with low precision multiplications

venue: Proceedings of the International Conference on Learning Representations (ICLR)

authors:
J. Deng
W. Dong
R. Socher
L.-J. Li
K. Li
L. Fei-Fei

title: Imagenet: A large-scale hierarchical image database

venue: 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee

authors:
J. Duchi
E. Hazan
Y. Singer

title: Adaptive subgradient methods for online learning and stochastic optimization

venue: Journal of machine learning research, 12(7)

authors:
I. Goodfellow
Y. Bengio
A. Courville

title: Deep learning

venue: MIT press Cambridge

authors:
A. Griewank
A. Walther

title: Evaluating derivatives: principles and techniques of algorithmic differentiation

venue: SIAM

authors:
S. Gupta
A. Agrawal
K. Gopalakrishnan
P. Narayanan

title: Deep learning with limited numerical precision

venue: International conference on machine learning, pages 1737–1746. PMLR

authors:
K. He
X. Zhang
S. Ren
J. Sun

title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

venue: Proceedings of the IEEE international conference on computer vision, pages 1026–1034

title: Deep residual learning for image recognition

venue: Proceedings of the IEEE conference on computer vision and pattern recognition

authors:
A.G. Howard
M. Zhu
B. Chen
D. Kalenichenko
W. Wang
T. Weyand
M. Andreetto
H. Adam

title: Mobilenets: Efficient convolutional neural networks for mobile vision applications

venue: arXiv preprint arXiv:1704.04861

authors:
K. Hwang
W. Sung

title: Fixed-point feedforward deep neural network design using weights+ 1

venue: 0, and- 1. In 2014 IEEE Workshop on Signal Processing Systems (SiPS), pages 1–6. IEEE

authors:
S. Ioffe
C. Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

venue: International conference on machine learning, pages 448–456. PMLR

authors:
D.P. Kingma
J. Ba

title: Adam: A method for stochastic optimization

venue: International Conference on Learning Representations, Conference Track Proceedings

authors:
A. Krizhevsky
V. Nair
G. Hinton

title: Cifar-10 (canadian institute for advanced research)

venue: URL http://www. cs. toronto. edu/kriz/cifar. html, 5

authors:
Y. LeCun
L. Bottou
Y. Bengio
P. Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE, 86(11)

authors:
Y. Netzer
T. Wang
A. Coates
A. Bissacco
B. Wu
A.Y. Ng

title: Reading digits in natural images with unsupervised feature learning

venue: NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011

authors:
A. Paszke
S. Gross
F. Massa
A. Lerer
J. Bradbury
G. Chanan
T. Killeen
Z. Lin
N. Gimelshein
L. Antiga
A. Desmaison
A. Kopf
E. Yang
Z. DeVito
M. Raison
A. Tejani
S. Chilamkurthy
B. Steiner
L. Fang
J. Bai
S. Chintala

title: Pytorch: An imperative style

venue: high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.

authors:
R.T. Rockafellar
R.J.-B. Wets

title: Variational analysis

venue: volume 317. Springer Science & Business Media

authors:
A. Rodriguez
E. Segal
E. Meiri
E. Fomenko
Y.J. Kim
H. Shen
B. Ziv

title: Lower numerical precision deep learning inference and training

venue: Intel White Paper, 3:1–19

authors:
D.E. Rumelhart
G.E. Hinton
R.J. Williams

title: Learning representations by backpropagating errors

title: Introduction to piecewise differentiable equations

venue: Springer Science & Business Media

authors:
K. Simonyan
A. Zisserman

title: Very deep convolutional networks for large-scale image recognition

venue: International Conference on Learning Representations

authors:
N. Srivastava
G. Hinton
A. Krizhevsky
I. Sutskever
R. Salakhutdinov

title: Dropout: a simple way to prevent neural networks from overfitting

venue: The journal of machine learning research, 15(1):1929–1958

authors:
V. Vanhoucke
A. Senior
M.Z. Mao

title: Improving the speed of neural networks on cpus

venue: Deep Learning and Unsupervised Feature Learning NIPS Workshop.

heading: 1 Introduction

text:
Nonsmooth algorithmic differentiation: The training phase of neural networks relies on firstorder methods such as Stochastic Gradient Descent (SGD) [14, 9] and crucially on algorithmic differentiation [15]. The fast “differentiator” used in practice to compute mini-batch descent directions is the backpropagation algorithm [29, 4]. Although designed initially for differentiable problems, it is applied indifferently to smooth or nonsmooth networks. In the nonsmooth case this requires surrogate derivatives at the non regularity points. We focus on the famous ReLU := max(0, ·), for which a value for s = ReLU′(0) has to be chosen. A priori, any value in [0, 1] bears a variational sense as it corresponds to a subgradient [27]. Yet in most libraries s = 0 is chosen; it is the case for TensorFlow [2], PyTorch [26] or Jax [10]. Why this choice? What would be the impact of a different value of s? How this interacts with other training strategies? We will use the notation backprops to denote backpropagation implemented with ReLU′(0) = s for any given real number s.1
1Definition in Section 2. This can be coded explicitly or cheaply emulated by backprop0. Indeed, considering fs : x 7→ ReLU(x) + s(ReLU(−x)− ReLU(x) + x) = ReLU(x), we have backprop0[fs](0) = s.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
What does backpropagation compute? A popular thinking is that the impact of s = ReLU′(0) should be limited as it concerns the value at a single point. This happens to be exact in theory but for surprisingly complex reasons related to Whitney stratification (see Section 2 and references therein):
— For a given neural network architecture, backprops outputs a gradient almost everywhere, independently of the choice of s ∈ R, see [7] and [5] for a detailed treatment of ReLU networks. — Under proper randomization of the initialization and the step-size of SGD, with probability 1 the value of s has no impact during training with backprops as a gradient surrogate, see [8, 6].
In particular, the set of network parameters such that backprop0 6= backprop1 is negligible and the vast majority of SGD sequences produced by training neural networks are not impacted by changing the value of s = ReLU′(0). These results should in principle close the question about the role of ReLU′(0). Yet, this is not what we observe in practice with the default settings of usual libraries.
A surprising experiment and the bifurcation zone: An empirical observation on MNIST triggered our investigations: consider a fully connected ReLU network, and let (θk,0)k∈N and (θk,1)k∈N be two training weights sequences obtained using SGD, with the same random initialization and the same mini-batch sequence but choosing respectively s = 0 and s = 1 in PyTorch [26]. As depicted in Figure 1 (left), the two sequences differ. A closer look shows that the sudden divergence is related to what we call the bifurcation zone, i.e., the set S0,1 of weights such that backprop0 6= backprop1. As mentioned previously this contradicts theory which predicts that the bifurcation zone is met with null probability during training. This contradiction is due to the precision of floating point operations and, to a lesser extent, to the size of deep learning problems. Indeed, rounding schemes used for inexact arithmetics over the reals (which set to zero all values below a certain threshold), may “thicken” negligible sets. This is precisely what happens in our experiments (Figure 1).
The role of numerical precision: Contrary to numerical linear algebra libraries such as numpy, which operates by default under 64 bits precision, the default choice in PyTorch is 32 bits precision (as in TensorFlow or Jax). We thus modulated machine precision to evaluate the importance of the bifurcation zone in Section 3. In 32 bits, we observed that empirically the zone occupies from about 10% to 90% of the network weight space. It becomes invisible in 64 bits precision even for quite large architectures, while, in 16 bits, it systematically fills up the whole space. Although numerical precision is the primary cause of the apparition of the zone, we identify other factors such as network size, sample size. Let us mention that low precision neural network training is a topic of active research [33, 20, 11, 16], see also [28] for an overview. Our investigations are complementary as we focus on the interplay between nonsmoothness and numerical precision.
Impact on learning: The next natural question is to measure the impact of the choice of s = ReLU′(0) in machine learning terms. In Section 4, we conduct extensive experiments combining different architectures (fully connected, VGG, ResNet), datasets (MNIST, SVHN, CIFAR10,
ImageNet) and other learning factors (Adam optimizer, batch normalization, dropout). In 32 bits numerical precision (default in PyTorch or Tensorflow), we consistently observe an effect of choosing s 6= 0. We observe a significant decrease in terms of test accuracy as |s| increases; this can be explained by chaotic oscillatory behaviors induced during training. In some cases gradients even explode and learning cannot be achieved. The sensitivity to this effect highly depends on the problem at hand, in particular, on the network structure and the dataset. On the other hand the choice s = 0 seems to be the most stable. We also observe that both batch normalization [21] and—to a lesser degree—the Adam optimizer [22] considerably mitigate this effect. All our experiments are done using PyTorch [26]; we provide the code to generate all figures presented in this manuscript.
One important message is that, even if the default choice s = 0 seems to be the most stable, our experiments show a counter-intuitive phenomenon that illustrates the interplay between numerical precision and nonsmoothness, and calls for caution when learning nonsmooth networks.
Outline of the paper: In Section 2 we recall elements of nonsmooth algorithmic differentiation which are key to understand the mathematics underlying our experiments. Most results were published in [7, 8]; we provide more detailed pointers to this literature in Appendix A.1. In Section 3 we describe investigations of the bifurcation zone and factors influencing its importance using fully connected networks on the MNIST dataset. Neural network training experiments are detailed in Section 4 with further experimental details and additional experiments reported in Appendix D.

heading: 2 On the mathematics of backpropagation for ReLU networks

text: This section recalls recent advances on the mathematical properties of backpropagation, with in particular the almost sure absence of impact of ReLU′(0) on the learning phase (assuming exact arithmetic over the reals). The main mathematical tools are conservative fields developed in [7]; we provide a simplified overview which is applicable to a wide class of neural networks.

heading: 2.1 Empirical risk minimization and backpropagation

text:
Given a training set {(xi, yi)}i=1...N , the supervised training of a neural network f consists in minimizing the empirical risk:
min θ∈RP
J(θ) := 1
N N∑ i=1 `(f(xi, θ), yi) (1)
where θ ∈ RP are the network’s weight parameters and ` is a loss function. The problem can be rewritten abstractly, for each i = 1, . . . , N and θ ∈ RP , `(f(xi, θ), yi) = li(θ) where the function li : RP → R is a composition of the form
li = gi,M ◦ gi,M−1 ◦ . . . ◦ gi,1 (2) where for each j = 1, . . . ,M , the function gi,j is locally Lipschitz with appropriate input and output dimensions. A concrete example of what the functions gi,j look like is given in Appendix A.2 in the special case of fully connected ReLU networks. Furthermore, we associate with each gi,j a generalized Jacobian Ji,j which is such that Ji,j(w) = Jacgi,j (w) whenever gi,j is differentiable at w and Jac denotes the usual Jacobian. The value of Ji,j at the nondifferentiability loci of gi,j can be arbitrary for the moment. The backpropagation algorithm is an automatized implementation of the rules of differential calculus: for each i = 1, . . . , N , we have
backprop li(θ) = Ji,M (gi,M−1 ◦ . . . ◦ gi,1(θ))× Ji,M−1 (gi,M−2 ◦ . . . ◦ gi,1(θ))× . . .× Ji,1(θ). (3)
Famous autograd libraries such as PyTorch [26] or TensorFlow [1] implement dictionaries of functions g with their corresponding generalized Jacobians J , as well as efficient numerical implementation of the quantities defined in (3).

heading: 2.2 ReLU networks training

text:
Our main example is based on the function ReLU: R→ R defined by ReLU(x) = max{x, 0}. It is differentiable save at the origin and satisfies ReLU′(x) = 0 for x < 0 and ReLU′(x) = 1 for x > 0.
The value of the derivative at x = 0 could be arbitrary in [0, 1] as we have ∂ReLU(0) = [0, 1], where ∂ denotes the subgradient from convex analysis. Let us insist on the fact that any value within [0, 1] has a variational meaning. For example PyTorch and TensorFlow use ReLU′(0) = 0.
Throughout the paper, and following the lines of [8], we say that a function g : Rp → Rq is elementary log-exp if it can be described by a finite compositional expression involving the basic operations +,−,×, / as well as the exponential and logarithm functions, inside their domains of definition. Examples include the logistic loss log(1 + e−x) on R, the multivariate Gaussian density (2π)−K/2 exp(− ∑K k=1 x 2 k/2) on RK , and the softmax function ( exi/ ∑K k=1 e xk ) 1≤i≤K on R
K . The expressions x3/x2 and exp(−1/x2) do not fit this definition because evaluation at x = 0 cannot be defined by the formula. Roughly speaking a computer evaluating an elementary log-exp expression should not output any NaN error for any input.
Definition 1 (ReLU network training). Assume that in (1), the function ` is elementary log-exp, the network f has an arbitrary structure and the functions gi,j in (2) are either elementary log-exp or consist in applying the ReLU function to some coordinates of their input. We then call the problem in (1) a ReLU network training problem. Furthermore, for any s ∈ R, we denote by backprops the quantity defined in (3) when ReLU′(0) = s for all ReLU functions involved in the composition (2).
Other nonsmooth activation functions: The ReLU operation actually allows to express many other types of nonsmoothness such as absolute values, maxima, minima, quantiles (med for median) or soft-thresholding (st). For any x, y, z ∈ R, we indeed have |x| = ReLU(2x) − x, 2 max(x, y) = (|x − y| + x + y), min(x, y) = −max(−x,−y), med(x, y, z) = min(max(x, y),max(x, z),max(y, z)), st(x) = ReLU(x− 1)− ReLU(−x− 1). Definition 1 is thus much more general than it may seem since it allows, for example, to express convolutional neural networks with max-pooling layers such as VGG or ResNet architectures which correspond to the models considered in the experimental section (although we do not re-program pooling using ReLU). The following theorem is due to [7], with an alternative version in [8].
Theorem 1 (Backprop returns a gradient a.e.). Consider a ReLU network training problem (1) as in Definition 1 and T ≥ 1. Define S ⊂ RP as the complement of the set{ θ ∈ RP , li differentiable at θ, backpropsli(θ) = ∇li(θ), ∀i ∈ {1, . . . , N}, s ∈ [−T, T ] } .
Then S is contained in a finite union of embedded differentiable manifolds of dimension at most

heading: P − 1 (and in particular has Lebesgue measure zero).

text: Although this theorem looks natural, this is a nontrivial result about the backpropagation algorithm that led to the introduction of conservative fields [7]. It implies that all choices for s = ReLU′(0) in [0, 1] = ∂ReLU(0) are essentially equivalent modulo a negligible set S. Perhaps more surprisingly, s can be chosen arbitrarily in R without breaking this essential property of backprop. The set S is called the bifurcation zone throughout the manuscript. For ReLU network training problems, the bifurcation zone is a Lebesgue zero set and is actually contained locally in a finite union of smooth objects of dimension strictly smaller than the ambient dimension. This geometric result reveals a surprising link between backpropagation and Whitney stratifications, as described in [7, 8]. In any case the bifurcation zone is completely negligible. Note that the same result holds if we allow each different call to the ReLU function to use different values for ReLU′(0).

heading: 2.3 ReLU network training with SGD

text:
Let (Bk)k∈N denote a sequence of mini-batches with sizes |Bk| ⊂ {1, . . . , N} for all k and αk > 0 the learning rate. Given initial weights θ0 ∈ RP and any parameter s ∈ R, the SGD training procedure of f consists in applying the recursion
θk+1,s = θk,s − γ αk |Bk| ∑ n∈Bk backprops[`(f(xn, θk,s), yn)], (4)
where γ > 0 is a step-size parameter. Note that we explicitly wrote the dependency of the sequence in s = ReLU′(0). According to Theorem 1 if the initialization θ0 is chosen randomly, say, uniformly in a ball, a hypercube, or with iid Gaussian entries, then with probability 1, θ0 does not fall in the bifurcation zone S. Intuitively, since S is negligible, the odds that one of the iterates produced by the
algorithm fall on S are very low. As a consequence, varying s in the recursion (4) does not modify the sequence. This rationale is actually true for almost all values of γ. This provides a rigorous statement of the idea that the choice of ReLU′(0) “does not affect” neural network training. The following result is based on arguments developed in [8], see also [6] in a probabilistic context. Theorem 2 (ReLU′(0) does not impact training with probability 1). Consider a ReLU network training problem as in Definition 1. Let (Bk)k∈N be a sequence of mini-batches with |Bk| ⊂ {1, . . . , N} for all k and αk > 0 the associated learning rate parameter. Choose θ0 uniformly at random in a hypercube and γ uniformly in a bounded interval I ⊂ R+. Let s ∈ R, set θ0,s = θ0, and consider the recursion given in (4). Then, with probability one, for all k ∈ N, θk,s = θk,0.

heading: 3 Surprising experiments on a simple feedforward network

heading: 3.1 ReLU′(0) has an impact

text:
Even though the ReLU activation function is non-differentiable at 0, autograd libraries such as PyTorch [26] or TensorFlow [1] implement its derivative with ReLU′(0) = 0. What happens if one chooses ReLU′(0) = 1? The popular answer to this question is that it should have no effect. Theorems 1 and 2 provide a formal justification which is far from being trivial.
A 32 bits MNIST experiment We ran a simple experiment to confirm this answer. We initialized two fully connected neural networks f0 and f1 of size 784×2000×128×10 with the same weights θ0,0 = θ0,1 ∈ RP which are chosen at random. Using the MNIST dataset [24], we trained f0 and f1 with the same sequence of mini-batches (Bk)k∈N (minibatch size 128), using the recursion in (4) for s = 0 and s = 1 and with a fixed αk = 1, and γ chosen uniformly at random in [0.01, 0.012]. At each iteration k, we computed the sum ‖θk,0 − θk,1‖1 of the absolute differences between the coordinates of θk,0 and θk,1. As a sanity check, we actually computed θk,0 a second time, denoting this by θ̃k,0, using a third network to control for sources of divergence in our implementation. Results are reported in Figure 1. The experiment was run using PyTorch [26] on a CPU.
ReLU’(0) has an impact First we observe no difference between θk,0 and θ̃k,0, which shows that we have controlled all possible sources of divergence in PyTorch. Second, while no differences between θ0,0 and θ0,1 is expected (Theorem 2), we observe a sudden deviation of ‖θk,0 − θk,1‖1 at iteration 65 which then increases in a smooth way. The deviation is sudden as the norm is exactly zero before iteration 65 and jumps above one after. Therefore this cannot be explained by an accumulation of small rounding errors throughout iterations, as this would produce a smooth divergence starting at the first iteration. So this suggests that there is a special event at iteration 65.
The center part of Figure 1 displays the minimal absolute value of neurons of the first hidden layer evaluated on the current mini-batch, before the application of ReLU. It turns out that at iteration 65, this minimal value is exactly 0, resulting in a drop in the center of Figure 1. This means that the divergence is actually due to an iterate of the sequence falling exactly on the bifurcation zone. According to Theorem 2, this event is so exceptional that it should never been seen.
Practice and Theory: Numerical precision vs Genericity This contradiction can be solved as follows: the minimal absolute value in Figure 1 oscillates between 10−6 and 10−8 which is roughly the value of machine precision in 32 bits float arithmetic. This machine precision value is of the order 10−16 in 64 bits floating arithmetic which is orders of magnitude smaller than the typical value represented in Figure 1. And indeed, performing the same experiment in 64 bits precision, the divergence of ‖θk,0 − θk,1‖1 disappears and the algorithm can actually be run for many epochs without any divergence between the two sequences. This is represented in Figure 7 in Appendix B. We also report similar results using ReLU6 [19] in place of ReLU on a similar network.

heading: 3.2 Relative volume of the bifurcation zone and relative gradient variation

text:
The previous experiment suggests that mini-batch SGD algorithm (4) crossed the bifurcation zone:
S01 = {θ ∈ RP : ∃i ∈ {1, . . . , N}, backprop0[li](θ) 6= backprop1[li](θ)} ⊂ S. (5) This unlikely phenomenon is due to finite numerical precision arithmetic which thickens the negligible set S01 in proportion to the machine precision threshold. This is illustrated on the right of
Figure 1, which represents the bifurcation zone at iteration 65 in a randomly generated hyperplane (uniformly among 2 dimensional hyperplanes) centered around the weight parameters which generated the bifurcation in the previous experiment (evaluation on the same mini-batch). The white area corresponds to some entries being exactly zero, i.e., below the 32 bits machine precision threshold, before application of ReLU. On the other hand, in 64 bits precision, the same representation is much smoother and does not contain exact zeros (see Figure 7 in Appendix B). This confirms that the level of floating point arithmetic precision explains the observed divergence. We now estimate the relative volume of this bifurcation zone by Monte Carlo sampling (see Appendix C for details). All experiments are performed using PyTorch [26] on GPU.
Experimental procedure – weight randomization: We randomly generate a set of parameters {θj}Mj=1, with M = 1000, for a fully connected network architecture f composed of L hidden layers. Given two consecutive layers, respectively composed of m and m′ neurons, the weights of the corresponding affine transform are drawn independently, uniformly in [−α, α] where α = √ 6/ √ m. This is the default weight initialization scheme in PyTorch (Kaiming-Uniform [17]). Given this sample of parameters, iterating on the whole MNIST dataset, we approximate the proportion of θj for which backprop0(li)(θj) 6= backprop1(li)(θj) for some i, for different networks and under different conditions (see Appendix C for details).
Impact of the floating-point precision: Using a fixed architecture of three hidden layers of 256 neurons each, we empirically measured the relative volume of S01 using the above experimental procedure, varying the numerical precision. Table 1 reports the obtained estimates. As shown in Table 1, line 1, at 16 bits floating-point precision, all drawn weights {θi}Mi=1 fall within S01. In sharp contrast, when using a 64 bits precision, none of the sampled weights belong to S01. This proportion is 40% in 32 bits precision. For the rare impacted mini-batches (Table 1 line 2), the relative change in norm is above a factor 20, higher in 16 bits precision (Table 1 line 3). These results confirm that the floating-point arithmetic precision is key in explaining the impact of ReLU′(0) during backpropagation, impacting both frequency and magnitude of the differences.
Impact of sample and mini-batch size: Given a training set {(xn, yn)}n=1...N and a random variable θ ∈ RP with distribution Pθ, we estimate the probability that θ ∈ S01 ⊂ S. Intuitively, this probability should increase with sample size N . We perform this estimation for a fixed architecture of 4 hidden layers composed of 256 neurons each while varying the sample size. Results are reported in Figure 2. For both the 16 and the 32 bits floating-point precisions, our estimation indeed increases with the sample size while we do not find any sampled weights in S01 in 64 bits precision. We also found that the influence of mini-batch size is not significative.
Impact of network size: To evaluate the impact of the network size, we carried out a similar Monte Carlo experiment, varying the depth and width of the network. Firstly, we fixed the number of hidden layers to 3. Following the same experimental procedure, we empirically estimated the relative volume of S01, varying the width of the hidden layers. The results, reported in Figure 2, show that increasing the number of neurons by layer increases the probability to find a random θ ∼ Pθ in S01 for both 16 and 32 floating-point precision. In 64 bits, even with the largest width tested (1024 neurons), no sampled weight parameter is found in S01. Similarly we repeated the experiment varying the network depth and fixing, this time, the width of the layers to 256 neurons. Anew, the
results, reported in Figure 2, show that increasing the network depth increases the probability that θ ∼ Pθ belongs to S01 for both 16 and 32 bits floating-point precision while this probability is zero in 64 bits precision. This shows that the size of the network, both in terms of number of layers and size of layers is positively related to the effect of the choice of s in backpropagation. On the other hand, the fact that neither the network depth, width, or the number of samples impact the 64 bits case suggests that, within our framework, numerical precision is the primary factor of deviation.

heading: 4 Consequences for learning

heading: 4.1 Benchmarks and implementation

text: Datasets and networks We further investigate the effect of the phenomenon described in Section 3 in terms of learning using the CIFAR10 dataset [23] and the VGG11 architecture [31]. To confirm our findings in alternative settings, we also use the MNIST [24], SVHN [25] and ImageNet [12] datasets, fully connected networks (3 hidden layers of size 2048), and the ResNet18 and ResNet50 architectures [18]. Additional details on the different architectures and datasets are found in Appendix D.1. By default, unless stated otherwise, we use the SGD optimizer. We also investigated the effect of batch normalization [21], dropout [32], the Adam optimizer [22] as well as numerical precision. All the experiments in this section are run in PyTorch [26] on GPU. For each training experiment presented in this section (except ImageNet experiments), we use the optuna library [3] to tune learning rates for each experimental condition; see also Appendix D.2.

heading: 4.2 Effect on training and test errors

text:
We first consider training a VGG11 architecture on CIFAR10 using the SGD optimizer. For different values of ReLU′(0), we train this network ten times with random initializations under 32 bits arithmetic precision. The results are depicted in Figure 3. Without batch normalization, varying the value of ReLU′(0) beyond a magnitude of 0.1 has a detrimental effect on the test accuracy, resulting in a concave shaped curve with maximum around ReLU′(0) = 0. On the other hand, the decrease of the training loss with the number of epochs suggests that choosing ReLU′(0) 6= 0 induces jiggling behaviors with possible sudden jumps during training. Note that the choice ReLU′(0) = 0 leads to a smooth decrease and that the magnitude of the jumps for other values is related to the magnitude of the chosen value. This is consistent with the interpretation that changing the value of ReLU′(0) has an excitatory effect on training. We observed qualitatively similar behaviors for a fully connected network on MNIST and a ResNet18 on CIFAR10 (Appendix D). Sensitivity to the magnitude of ReLU′(0) depends on the network architecture: our fully connected network on MNIST is less sensitive to this value than VGG11 and ResNet18. The latter shows a very high sensitivity since for values above 0.2, training becomes very unstable and almost impossible.
These experiments complement the preliminary results obtained in Section 3. In particular choosing different values for ReLU′(0) has an effect, it induces a chaotic behavior during training which affects test accuracy. The default choice ReLU′(0) = 0 seems to provide the best performances.
To conclude, we conducted four training experiments for ResNet50 on the ImageNet dataset [12] using the SGD optimizer. These were conducted with fixed learning rate, contrary to results reported above. We observe that switching from ReLU′(0) = 0 to 1 results in a massive drop from around 75% to 63% or 55% for two runs.

heading: 4.3 Mitigating factors: numerical precision, batch-normalization and Adam

text:
We analyze below the combined effects of the variations of ReLU′(0) with numerical precision or classical reconditioning methods: Adam, batch-normalization and dropout.
Batch-normalization: As represented in Figure 3, batch normalization [21] not only allows to attain higher test accuracy, but it also completely filters out the effect of the choice of the value of ReLU′(0), resulting in a flat shaped curve for test accuracy. This is consistent with what we observed on the training loss (Figure 12 in Appendix D.3) for which different choices of ReLU′(0) lead to indistinguishable training loss curves. This experiment suggests that batch normalization has a significant impact in reducing the effect of the choice of ReLU′(0). This observation was confirmed with a very similar behavior on the MNIST dataset with a fully connected network (Figure 9 in Appendix D.2). We could observe a similar effect on CIFAR 10 using a ResNet18 architecture (see Appendix D.5), however in this case the value of ReLU′(0) still has a significative impact on test error, the ResNet18 architecture being much more sensitive.
Using the Adam optimizer: The Adam optimizer [22] is among the most popular algorithms for neural network training; it combines adaptive step-size strategies with momentum. Adaptive stepsize acts as diagonal preconditioners for gradient steps [22, 13] and therefore can be seen as having an effect on the loss landscape of neural network training problems. We trained a VGG11 network on both CIFAR 10 and SVHN using the Adam optimizer. The results are presented in Figure 4. We observe a qualitatively similar behavior as in the experiments of Section 4.2 but a much lower sensitivity to the magnitude of ReLU′(0). In other words, the use of the Adam optimizer mitigates the effect of this choice, both in terms of test errors and by buffering the magnitude of the sometimes chaotic effect induced on training loss optimization (Figure 13 in Appendix D.3).
Increasing numerical precision: As shown in Appendix D.3, using 64 bits floating precision on VGG11 with CIFAR10 cancels out the effect of ReLU′(0) = 1, in coherence with Section 3. More specifically ReLU′(0) = 1 in 64 bits precision obtains similar performances as ReLU′(0) = 0 in 32 bits precision. Furthermore, the numerical precision has barely any effect when ReLU′(0) = 0. Finally, we remark that in 16 bits with ReLU′(0) = 1, training is extremely unstable so that we were not able to train the network in this setting.
Combination with dropout: Dropout [32] is another algorithmic process to regularize deep neural networks. We investigated its effect when combined with varying choices of ReLU′(0). We used a fully connected neural network trained on the MNIST dataset with different values of dropout probability. The results are reported in Figure 11 in Appendix D.2. We did not observe any joint effect of dropout probability and magnitude of ReLU′(0) in this context.

heading: 4.4 Back to the bifurcation zone: more neural nets and the effects of batch-norm

text:
The training experiments are complemented by a similar Monte Carlo estimation of the relative volume of the bifurcation zone as performed in Section 3 (same experimental setting). To avoid random outputs we force the GPU to compute convolutions deterministically. Examples of results are given in Figure 5. Additional results on fully connected network with MNIST and ResNet18 on CIFAR10 are shown in Section D. We consistently observe a high probability of finding an example on the bifurcation zone for large sample sizes. Several comments are in order.
Numerical precision: Numerical precision is the major factor in the thickening of the bifurcation zone. In comparison to 32 bits experiments, 16 bits precision dramatically increases its relative importance. We also considered 64 bits precision on SVHN, a rather large dataset. Due to the computational cost, we only drew 40 random weights and observed no bifurcation on any of the terms of the loss whatsoever. This is consistent with the experiments conducted in Section 3 and suggests that, within our framework, 64 bit precision is the main mitigating factor for our observations.
Batch normalization: In all our relative volume estimation experiments, we observe that batch normalization has a very significant effect on the proportion of examples found in the bifurcation zone. In 32 bits precision, the relative size of this zone increases with the addition of batch normalization, similar observations were made in all experiments presented in Appendix D. This is a counter-intuitive behavior as we have observed that batch normalization increases test accuracy and mitigates the effect of ReLU′(0). Similarly in 16 bits precision, the addition of batch normalization seems to actually decrease the size of the bifurcation zone. Batch normalization does not result in the same qualitative effect depending on arithmetic precision. These observations open many questions which will be the topic of future research.
4.5 Total number of ReLU calls during training
We consider the MNIST dataset with a fully connected ReLU network, varying the number of layers in 1, . . . , 6 and neuron per layers in 16, 64, 256, 512. For 16 and 32 bits precisions, we perform 100
epochs and proportion of ReLU(0) over all ReLU calls during training. Figure 6 suggests that, for a given precision, the number of times the bifurcation zone is met is roughly proportional to the total number of ReLU calls during training, independently of the architecture used (number of layers, neurons per layers). This suggests that the total number of ReLU calls during training is an important factor in understanding the bifurcation phenomenon. Broader investigation of this conjecture will be a matter of future work.

heading: 5 Conclusions and future work

text:
The starting point of our work was to determine if the choice of the value s = ReLU′(0) affects neural network training. Theory tells that this choice should have negligible effect. Performing a simple learning experiment, we discovered that this is false in the real world and the first purpose of this paper is to account for this empirical evidence. This contradiction between theory and practice is due to finite floating point arithmetic precision while idealized networks are analyzed theoretically within a model of exact arithmetic on the field of real numbers. Owing to the size of deep learning problems, rounding errors due to numerical precision occur at a relatively high frequency, and virtually all the time for large architectures and datasets under 32 bit arithmetic precision (the default choice for TensorFlow and PyTorch libraries).
Our second goal was to investigate the impact of the choice of s = ReLU′(0) in machine learning terms. In 32 bits precision it has an effect on test accuracy which seems to be the result of inducing a chaotic behavior in the course of empirical risk minimization. This was observed consistently in all our experiments. However we could not identify a systematic quantitative description of this effect; it highly depends on the dataset at hand, the network structure as well as other learning parameters such as the presence of batch normalization and the use of different optimizers. Our experiments illustrate this diversity. We observe an interesting robustness property of batch normalization and the Adam optimizer, as well as a high sensitivity to the network structure.
Overall, the goal of this work is to draw attention to an overlooked factor in machine learning and neural networks: nonsmoothness. The ReLU activation is probably the most widely used nonlinearity in this context, yet its nondifferentiability is mostly ignored. We highlight the fact that the default choice ReLU′(0) = 0 seems to be the most robust, while different choices could potentially lead to instabilities. For a general nonsmooth nonlinearity, it is not clear a priori which choice would be the most robust, if any, and our investigation underlines the potential importance of this question. Our research opens new directions regarding the impact of numerical precision on neural network training, its interplay with nonsmoothness and its combined effect with other learning factors, such as network architecture, batch normalization or optimizers. The main idea is that mathematically negligible factors are not necessarily computationally negligible.
Acknowledgments and Disclosure of Funding
The authors thank anonymous referees for constructive suggestions which greatly improved the paper. The authors acknowledge the support of the DEEL project, the AI Interdisciplinary Institute ANITI funding, through the French “Investing for the Future – PIA3” program under the Grant agreement ANR-19-PI3A-0004, Air Force Office of Scientific Research, Air Force Material Command, USAF, under grant numbers FA9550-19-1-7026, FA9550-18-1-0226, and ANR MaSDOL 19-CE23-0017-01. J. Bolte also acknowledges the support of ANR Chess, grant ANR-17-EURE0010 and TSE-P.

title: Numerical influence of ReLU’(0) on backpropagation


summaries: This paper introduces a novel approach to learning parameterized policies. The way I understand the work: it jointly optimizes a policy objective, a probabilistic dynamic model objective, and a compression objective. A bottleneck encoding distribution is key here, which is subject to both model predictions (given previous action and encoding) and compression (given current observation). In spirit, it is similar to variational information bottleneck (VIB) but can be seen as an extension of that to RL. So, the interpretation of this optimization is that the agent is aiming for encoding distribution that is maximally predictive of the next observation and maximally compressive of the observation while the agent is maximizing the future aggregate reward. The idea is neat. The optimization can also be easily put together. Some theoretical and empirical results are given.

summaries: In this paper, the authors propose a new reinforcement learning method, called robust predictable control, that optimizes a policy under limited representational resources. Similar to previous methods, they formalize the problem in a fully observable MDP and learn a compressed representation of the state space using a variational information bottleneck constraint. Crucially, they apply the bottleneck on sequences of states. Through a clever factorization of the variational prior, they obtain a prior dynamics model in the compressed state space. This dynamics model is predictive and the agent only needs additional representational resources if the encoded states are not predicted by the prior dynamics already. Using a set of standard RL benchmarks, the authors show that this set up improves compression and enforces predictable and robust behaviors .

summaries: This paper proposes the method "robust predictable control (RPC)". It proposes to use a information bottleneck between inputs and actions. However, in contrast to previous works, it does so not on a per-observation basis, but on a per-sequence basis using a latent model. Similar to some of the prior work, the encoding cost is not added to the loss, but subtracted from the reward.

summaries: The paper "Robust Predictable Control" describes a method to learn policies with an information bottleneck between environment observations and a latent representation which is used for decision-making. The bottleneck is posed between *sequences* of observation and latent states, in contrast to prior work, where a bottleneck is posed between individual states and latents independently. A crucial component of the proposed method is an action-conditioned dynamics model in the latent space. The proposed training objective is jointly minimized by training an encoder, latent dynamics model, and policy.  First, this encourages the encoder and dynamics model to yield good predictions of encoded environment observations.  Furthermore, by training the policy, the expected task return is maximized while ensuring that the behaviour of the agent is well-predictable.  The latent dynamics model formulation can be exploited to derive theoretical guarantees between open-loop and information-constrained closed-loop control.  Several experiments show that the information-constrained agent performs well in the nominal case but also on distorted environments and offers smooth transitions between different behaviours depending on the tightness of the information constraint.

abstractText: Many of the challenges facing today’s reinforcement learning (RL) algorithms, such as robustness, generalization, transfer, and computational efficiency, are closely related to compression. Prior work has convincingly argued why minimizing information is useful in the supervised learning setting, but standard RL algorithms lack an explicit mechanism for compression. The RL setting is unique because (1) its sequential nature allows an agent to use past information to avoid looking at future observations and (2) the agent can optimize its behavior to prefer states where decision making requires few bits. We take advantage of these properties to propose a method (RPC) for learning simple policies. This method brings together ideas from information bottlenecks, model-based RL, and bits-back coding into a simple and theoretically-justified algorithm. Our method jointly optimizes a latent-space model and policy to be self-consistent, such that the policy avoids states where the model is inaccurate. We demonstrate that our method achieves much tighter compression than prior methods, yielding up to 5× higher reward than a standard information bottleneck. As a result of this compression, the policies learned by our method are robust and generalize well to new tasks.1

authors:
A. Achille
S. Soatto

title: Emergence of invariance and disentanglement in deep representations

venue: The Journal of Machine Learning Research,

authors:
A.A. Alemi
I. Fischer
J.V. Dillon
K. Murphy

title: Deep variational information bottleneck

venue: In 5th International Conference on Learning Representations,

authors:
S. Arora
R. Ge
B. Neyshabur
Y. Zhang

title: Stronger generalization bounds for deep nets via a compression approach

venue: In International Conference on Machine Learning,

authors:
R. Bassily
S. Moran
I. Nachum
J. Shafer
A. Yehudayoff

title: Learners that use little information

venue: In Algorithmic Learning Theory,

authors:
X. Chen
K. He

title: Exploring simple siamese representation learning

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
K. Cho
B. van Merrienboer
Ç. Gülçehre
D. Bahdanau
F. Bougares
H. Schwenk
Y. Bengio

title: Learning phrase representations using RNN encoder-decoder for statistical machine translation

venue: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing,

authors:
J. Chung
K. Kastner
L. Dinh
K. Goel
A.C. Courville
Y. Bengio

title: A recurrent latent variable model for sequential data

venue: Advances in neural information processing systems,

authors:
K. Cobbe
O. Klimov
C. Hesse
T. Kim
J. Schulman

title: Quantifying generalization in reinforcement learning

authors:
J. Farebrother
M.C. Machado
M. Bowling

title: Generalization and regularization in dqn

venue: arXiv preprint arXiv:1810.00123

authors:
J. Foerster
G. Farquhar
M. Al-Shedivat
T. Rocktäschel
E. Xing
S. Whiteson

title: Dice: The infinitely differentiable monte carlo estimator

authors:
B.J. Frey
G.E. Hinton

title: Efficient stochastic source coding and an application to a bayesian network source model

authors:
C. Gelada
S. Kumar
J. Buckman
O. Nachum
M.G. Bellemare

title: Deepmdp: Learning continuous latent space models for representation learning

authors:
A. Goyal
R. Islam
D. Strouse
Z. Ahmed
H. Larochelle
M. Botvinick
S. Levine
Y. Bengio

title: Transfer and exploration via the information bottleneck

venue: In International Conference on Learning Representations

authors:
S. Guadarrama
A. Korattikara
O. Ramirez
P. Castro
E. Holly
S. Fishman
K. Wang
E. Gonina
N. Wu
E. Kokiopoulou
L. Sbaiz
J. Smith
G. Bartók
J. Berent
C. Harris
V. Vanhoucke
E. Brevdo

title: TF-Agents: A library for reinforcement learning in tensorflow. https://github.com/ tensorflow/agents. [Online; accessed 25-June-2019

authors:
S. Gui
H.N. Wang
H. Yang
C. Yu
Z. Wang
J. Liu

title: Model compression with adversarial robustness: A unified optimization framework

venue: Advances in Neural Information Processing Systems,

authors:
T. Haarnoja
A. Zhou
P. Abbeel
S. Levine

title: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

title: The cma evolution strategy: a comparing review

venue: Towards a new evolutionary computation,

authors:
G.E. Hinton
D. Van Camp

title: Keeping the neural networks simple by minimizing the description length of the weights

venue: In Proceedings of the sixth annual conference on Computational learning theory,

authors:
S.H. Huang
N. Papernot
I.J. Goodfellow
Y. Duan
P. Abbeel

title: Adversarial attacks on neural network policies

authors:
M. Igl
K. Ciosek
Y. Li
S. Tschiatschek
C. Zhang
S. Devlin
K. Hofmann

title: Generalization in reinforcement learning with selective noise injection and information bottleneck

authors:
P. Kamalaruban
Y. Huang
Y. Hsieh
P. Rolland
C. Shi
V. Cevher

title: Robust reinforcement learning via adversarial training with langevin dynamics

venue: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems

authors:
N.R. Ke
A. Singh
A. Touati
A. Goyal
Y. Bengio
D. Parikh
D. Batra

title: Learning dynamics model in reinforcement learning by incorporating the long term future

venue: arXiv preprint arXiv:1903.01599

venue: In ICLR (Poster)

authors:
D.P. Kingma
M. Welling

title: Auto-encoding variational bayes

venue: 2nd International Conference on Learning Representations,

authors:
M. Laskin
A. Srinivas
P. Abbeel

title: Curl: Contrastive unsupervised representations for reinforcement learning

authors:
Y. LeCun
J.S. Denker
S.A. Solla

title: Optimal brain damage

venue: In Advances in neural information processing systems,

authors:
A. Lee
A. Nagabandi
P. Abbeel
S. Levine

title: Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model

title: An environment for autonomous driving decision-making. https://github.com/ eleurent/highway-env

authors:
N. Littlestone
M. Warmuth

title: Relating data compression and learnability

authors:
X. Lu
K. Lee
P. Abbeel
S. Tiomkin

title: Dynamics generalization via information bottleneck in deep reinforcement learning

venue: arXiv preprint arXiv:2008.00614

authors:
A. Madry
A. Makelov
L. Schmidt
D. Tsipras
A. Vladu

title: Towards deep learning models resistant to adversarial attacks

venue: In 6th International Conference on Learning Representations,

authors:
J. Morimoto
K. Doya

title: Robust reinforcement learning

venue: Neural computation,

authors:
O. Nachum
S. Gu
H. Lee
S. Levine

title: Near-optimal representation learning for hierarchical reinforcement learning

venue: In 7th International Conference on Learning Representations,

authors:
A.Y. Ng
M.I. Jordan

title: PEGASUS: A policy search method for large mdps and pomdps

venue: UAI ’00: Proceedings of the 16th Conference in Uncertainty in Artificial Intelligence,

authors:
Oord
A. v. d
Y. Li
O. Vinyals

title: Representation learning with contrastive predictive coding

venue: arXiv preprint arXiv:1807.03748

authors:
P.A. Ortega
D.A. Braun

title: Thermodynamics as a theory of decision-making with informationprocessing costs

venue: Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences,

title: Learning complex, extended sequences using the principle of history compression

venue: Neural Computation,

authors:
O. Shamir
S. Sabato
N. Tishby

title: Learning and generalization with the information bottleneck

venue: Theoretical Computer Science,

authors:
C. Tessler
Y. Efroni
S. Mannor

title: Action robust reinforcement learning and applications in continuous control

authors:
N. Tishby
D. Polani

title: Information theory of decisions and actions

venue: In Perception-action cycle,

authors:
N. Tishby
N. Zaslavsky

title: Deep learning and the information bottleneck principle

venue: In 2015 IEEE Information Theory Workshop (ITW),

authors:
J. Veness
M.G. Bellemare
M. Hutter
A. Chua
G. Desjardins

title: Compress and control

venue: In Twenty-Ninth AAAI Conference on Artificial Intelligence

authors:
R. Wang
X. He
R. Yu
W. Qiu
B. An
Z. Rabinovich

title: Learning efficient multi-agent communication: An information bottleneck approach

authors:
D. Yarats
I. Kostrikov
R. Fergus

title: Image augmentation is all you need: Regularizing deep reinforcement learning from pixels

authors:
S. Ye
K. Xu
S. Liu
H. Cheng
Lambrechts
J.-H
H. Zhang
A. Zhou
K. Ma
Y. Wang
X. Lin

title: Adversarial robustness vs. model compression, or both

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

heading: 1 Introduction

text:
Many areas of reinforcement learning (RL) research focus on specialized problems, such as learning invariant representations, improving robustness to adversarial attacks, improving generalization, or building better world models. These problems are often symptoms of a deeper underlying problem: autonomous agents use too many bits from their environment. For the purpose of decision making, most information about the world is irrelevant. For example, a lane keeping feature on a car may take as input high-resolution camera input (millions of bits), but only needs to extract a few bits of information about the relative orientation of the car in the lane. Agents that rely on more bits of information run the risk of overfitting to the training task.
Agents that use few bits of information enjoy a number of appealing properties. These agents can better cope with high-dimensional sensory inputs (e.g., dozens of cameras on a self-driving car) and will be forced to learn representations that are more broadly applicable. Agents that throw away most information will be agnostic to idiosyncrasies in observations, providing robustness to missing or corrupted observations and allowing for transfer to different scenarios. For example, if an agent ignores 99.9% of bits, then corrupting a random bit is unlikely to change the agent’s behavior. Moreover, an agent that minimizes bits will prefer states where the dynamics are easy to predict, meaning that the agent’s resulting behavior will be easier to model. Thus, compression not only changes an agent’s representation, but also changes its behavior: an agent that can only use a limited number of bits will avoid risky behaviors that require more bits to execute (see Fig. 3a).
The generalization and robustness of a machine learning model is directly related to the complexity of that model. Indeed, standard techniques for reducing complexity, such as the information bottleneck [1, 43], can be directly applied to the RL setting [14, 21, 31, 42]. While these approaches make
1Project site with videos and code: https://ben-eysenbach.github.io/rpc
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
the policy’s action a simple function of the state, they ignore the temporal dimension of decision making. Instead, we will focus on learning policies whose temporally-extended behavior is simple, where we use compression to measure simplicity. Our key observation is that a policy’s behavior is simple if it is predictable.
Our method improves upon prior methods that apply an information bottleneck to RL [14, 21, 31] by recognizing two important properties of the decision making setting. First, because agents make a sequence of decisions, they can use salient information at one time step to predict salient information at the next time step. These predictions can decrease the amount of information that the agent needs to sense from the environment. We will show that learning a predictive model is not an ad-hoc heuristic, but rather a direct consequence of minimizing information using bits-back coding [12, 19]. Second, unlike supervised learning, the agent can change the distribution over states, choosing behaviors that visit states that are easier to compress. For example, imagine driving on a crowded road. Aggressively passing and tailgating other cars may result in reaching the destination faster, but requires careful attention to other vehicles and fast reactions. In contrast, a policy optimized for using few bits would not pass other cars and would leave a larger following distance (see Fig. 3b). Combined, these two capabilities result in a method that jointly trains a latent space model and a control policy, with the policy being rewarded for visiting states where that model is accurate. Unlike typical model-based methods, our method explicitly optimizes for the accuracy of open-loop planning and results in a model and policy that are self-consistent.
The main contribution of this paper is an RL algorithm, robust predictable control (RPC), for learning policies that use few bits of information. We will refer to such policies as compressed policies. RPC brings together ideas from information bottlenecks, model-based RL, and bits-back coding into a simple and theoretically-justified algorithm. RPC attains up to 5× higher return than a standard information bottleneck when compared at the same bitrate. Experiments demonstrate that the compressed policies learned by RPC are more robust than those learned by alternative approaches, generalize well to new tasks, and learn behaviors that can be composed for hierarchical RL.

heading: 2 Related Work

text:
Our work builds on a large body of prior work, both in supervised learning and reinforcement learning, that argues that compression is a good objective for learning representations and improving robustness [4, 19, 27, 30, 38, 39, 42, 44]. We measure compression by the mutual information between its inputs and outputs [4, 43], a metric that has been used to study the generalization properties [43] and representations learned by neural networks [1]. Our work extends these results to the RL setting by observing that the agent can change its behavior (i.e., the data distribution) to be more easily compressed. In the RL community, prior work has used the variational information bottleneck (VIB) [2] to minimize communication between agents in a multi-agent setting [45] and to improve exploration in a goal-reaching setting [14]. The most related RL methods are those that use an information bottleneck in RL to improve generalization [21, 31]. Whereas these prior methods compress observations individually, we will compress sequences of observations, similar to a sequential VAE [8, 23]. This difference, which corresponds to learning a latent-space model, improves compression and increases robustness on downstream tasks.
Prior work in RL has sought robustness, transfer, and generalization in many ways. For example, prior RL methods often use explicit representation learning objectives to accelerate learning of image-based tasks [13, 26, 28, 36]. The presentations learned by our method, like those learned by prior contrastive learning methods [26, 34, 36], do not require reconstructing observations. Prior work has also studied numerous strategies for learning policies that are resilient to perturbations in the environment, a problem known as robust RL [20, 22, 33, 41]. While prior robust RL methods typically involve solving a two-player game, we show that compression is a simpler mechanism for achieving some robustness benefits. Finally, the problem of learning RL policies that generalize has been studied by many prior papers [9, 10].

heading: 3 Reinforcement Learning with Fewer Bits

text: This section introduces the idea that predicting the future allows RL policies to operate with fewer bits. We derive this idea from first principles, develop it into a complete RL method, then discuss connections with model-based RL, bits-back coding, and other related topics.

heading: 3.1 Notation and Preliminaries

text:
An agent interacts in an MDP defined by states st and actions at; we will use “states” and “observations” interchangeably. The agent samples actions from a policy πθ(at | st, st−1, st−2, · · · ).2 We will construct this policy by learning an encoder φ(zt | st) (which encodes a state st into a representation zt), and a high-level policy πzθ(at | zt), which decodes a representation zt into an action at. The environment dynamics are defined by an initial state s1 ∼ p1(s1) and a transition function p(st+1 | st,at). The RL objective is to maximize the expected γ-discounted sum of rewards: maxπ Eπ [ ∑∞ t=1 γ
tr(st,at)]. The discount factor can be viewed as saying that the episode terminates at every time step with probability (1− γ), an interpretation we will use in Sec. 3.2. A model is simpler if it expresses a simpler input-output relationship [4, 43]. We will measure the complexity of a function using an information bottleneck, which is the mutual information I(x;y) between an input x and an output y [1, 43]. The variational information bottleneck provides a tractable upper bound on mutual information [1, 2]:
I(x;y) ≤ Ep(x,y) [ log ( p(y | x) m(y) )] ,
where m(y) is an arbitrary prior distribution. Applying the information bottleneck to an intermediate layer z = φ(x) is sufficient for bounding the mutual information between input x and output y.
Following prior work on compression in RL [21, 31], we aim to maximize rewards while minimizing the number of bits. While there are many ways to apply compression to RL (e.g., compressing actions, goals, or individual observations), we will focus on compressing sequences of states. The input is a sequence of states, s1:∞ , (s1, s2, · · · ); the output is a sequence of actions, a1:∞ , (a1,a2, · · · ). The objective is to learn a representation φθ(zt | st) and policy πθ(at | zt) that maximize reward, subject to the constraint that the policy uses on average C > 0 bits of information per episode:
max θ
Eπ,φ [ ∞∑ t=1 γtr(st,at) ] s.t. Eπ[I(s1:∞; z1:∞)] ≤ C, (1)
While prior work on compression in RL [21, 31] has applied the VIB to states independently, we will aim to compress entire sequences of observations. Applying the VIB to sequences allows us to use more expressive choices of the prior m(z1:∞). Our prior will use previous representations to predict the prior for the next representation, allowing us to obtain a tighter bound on mutual information.

heading: 3.2 Using Fewer Bits by Predicting the Future

text:
Predicting the future allows agents to observe fewer bits from future observations. To further decrease the number of bits used, the agent can change its behavior to visit states that are more easily compressed. States where the dynamics are hard to predict will require more bits, so the agent will prefer visiting states where its learned model can accurately predict the next state.
This intuition corresponds to solving the optimization problem in Eq. 1 with a prior that is factored autoregressively: m1:∞(z1:∞ | a1:∞) = m(z1) ∏ tmθ(zt+1 | zt,at). Note that the prior has learnable parameters θ. We apply the VIB to obtain an upper bound on the constraint in Eq. 1:
Eπ[I(s1:∞; z1:∞)] ≤ Eπ [ log ( p(z1:∞ | s1:∞) m(z1:∞) )] = Eπ [ ∞∑ t=1 γt(log φθ(zt | st)− logmθ(zt+1 | zt,at)) ] . (2) This objective is different from prior work that applies the VIB to RL [14, 21, 31] because the prior mθ(zt) is predicted, rather than fixed to be a unit Normal distribution. The discount factor γ reflects the assumption from Sec. 3.1 that the episode terminates with probability (1− γ) at each time step; of course, no bits are used after the episode terminates. Our final objective optimizes the policy π, the encoder φ, and the prior m to maximize reward and minimizing information:
max θ Eπθ,φθ,mθ [ ∞∑ t=1 γtr(st,at) ] s.t. Eπθ,φθ,mθ [∑ t γt (log φθ(zt | st)− logmθ(zt+1 | zt,at)) ] ≤ C. (3) For the encoder, this objective looks like a modification of the information bottleneck where the prior is predicted from the previous representation and action. Of course, we cannot predict the first
2We condition the policy on sequences of states because, even if the task is Markovian, the agent’s choice to omit certain bits from the representation can turn an otherwise fully-observed task into a partially-observed task.
representation, so we substitute use a unit Normal distribution for the prior at initial time step. For the policy, this objective looks like the standard RL objective but with a different reward function, one that includes an additional information cost:
r̃λ(st,at, st+1) , r(st,at) + λ (logmθ(zt+1 | zt,at)− log φθ(zt+1 | st+1))︸ ︷︷ ︸ (negative of the) information cost , (4)
where λ is the cost per bit. The term log φθ(zt | st) corresponds to the number of bits required to represent the representation zt. The term logmθ(zt+1 | zt,at) reflects how well the agent can predict the next representation. In effect, the agent has to “pay” for bits of the observation, but it gets a “refund” on bits that it predicted from the previous time step, analogous to bits-back coding [12, 19]. Said in other words, we want to minimize the excess bits required to infer zt from the current state, relative to the number of bits required to predict zt from the previous (zt−1,at−1). Note that the standard VIB approaches to RL [14, 21, 31] do not receive this refund because they compress observations independently, rather than compressing sequences of observations.
Note that the policy, encoder, and model are all jointly optimized with respect to the same objective, so updates to any component increases the objective for all components. Importantly, the agent optimizes not only its representation but also its behavior to minimize this information cost: the agent learns a representation that is easily predictable and learns to visit states where that representation is easily predictable. We therefore call our method robust predictable control (RPC).
4 A Practical Algorithm
Our method for optimizing (Eq. 3) is an actor-critic method applied to the informationaugmented reward function in Eq. 4. We introduce a Q function,
Qψ(st,at) = E [∑
t
γtr̃(st,at, st+1) ] ,
corresponding to the augmented reward function r̃. We optimize this Q function using standard temporal difference learning:
L(ψ) = 1 2 (Qψ(st,at)− yt)2 (5)
where yt = br̃t + γQψ(st+1,at+1)csg and b·csg is the stop-gradient operator [6, 11]. Since our overall objective (Eq. 1) only entails compressing the policy, not the Q function, we condition the Q function directly on the state. To optimize the encoder, prior, and policy, we reexpress our objective (Eq. 1) in terms of the immediate reward plus the Q function (see derivation in Appendix A.1):
L(θ; st) = Ezt−1∼φθ(st−1),st∼p(st|st−1,at−1) zt∼φ(st),at∼πzθ (at|zt) [Qψ(st,at)+λ (logmθ(zt | zt−1,at−1)− log φθ(zt | st))]. (6) Since the encoder is stochastic, we compute gradients using the reparametrization trick [25]. The fact that all three components are jointly optimized with respect to the same objective makes implementation of RPC surprisingly simple. Unlike sequential VAEs and RNNs, training agents with RPC does not require sampling frommθ(zt+1 | zt,at) and does not require backpropagation through time. In our implementation, we instantiate the encoder φθ(st), the prior mθ(zt+1 | zt,at), and the high-level policy πzθ(at | zt) as neural networks with parameters θ. The Q function Qψ(st,at) is likewise represented as a neural network with parameters φ. We update the dual parameter λ ≥ 0 using dual gradient descent. We use standard tricks such as target networks and taking the minimum over two Q values. We refer the reader to Appendix B and the open-sourced code for details.

heading: 5 Connections and Analysis

text: In this section we discuss the connections between robust predictable control and other areas of RL. We then prove that RPC enjoys certain robustness guarantees.

heading: 5.1 Connections

text:
This section explains connections between RPC and related ideas in the RL literature, with the aim of building intuition into how RPC works and providing an explanation for why RPC should learn robust policies and useful representations. We include a further discussion of the relationship to MaxEnt RL, the expression for the optimal encoder, and the value of information in Appendix A.3.
Model-Based RL. The prior mθ(zt | zt−1,at−1) learned by RPC can be viewed as a dynamics model. Rather than predicting what the next state will be, this model predicts what the representation of the next state will be. While the model is trained to make accurate predictions, the policy is also trained to visit states and actions where the model will be more accurate.
Representation learning. What, precisely, does the representation zt represent? Given a prior m(zt+1 | zt,at) and latent-conditioned policy πz(at | zt), we can use the current representation zt to predict good actions at both the current time step and (by unrolling the prior) at future time steps. Thus, the representation zt can be thought of as a compact representation of open-loop action sequences. Our experiments demonstrate that this compact representation of action sequences, once learned on one task, can be used to quickly learn a range of downstream tasks.
We can also view RPC as learning a new action space for the MDP, where the reward function is now r̃ (Eq. 4). What we previously called the encoder, φ(zt | st), is now the policy for selecting the actions zt in this new MDP. The − log φ(zt | st) term in from r̃ is the entropy of this encoder, so the encoder is performing MaxEnt RL on this new MDP. This new MDP encodes a strong prior for open-loop policies: sampling actions from the prior yields high reward.
Open-loop control. RPC learns a model m(zt | zt−1,at−1) that predicts the state representation at the next time step. Thus, we can unroll our policy in an open-loop manner, without observing transitions from the true system dynamics. Because the model and policy are trained to be selfconsistent, we expect that the highly compressed policies learned by RPC will perform well in this open-loop setting, as compared to uncompressed policies (see experiments in Sec. 6).

heading: 5.2 Theoretical Guarantees

text:
We now present analysis showing that the compressed policies learned by RPC generalize better. We also discuss a close connection between compression and open-loop control, proving that RPC learns policies and models that are self-consistent in a way that guarantees good performance when even when the agent cannot observe new observations. We do not intend to to analyze all benefits deriving from model compression (see, e.g., [3, 4]). All proofs are in Appendix C.
In supervised learning, it is well known that compression results in models that generalize better from the training set to the testing set. Our main result applies this same reasoning to RL. We assume that the policy has been trained on an empirical distribution of MDPs, and will be evaluated on a new MDP sampled from that sample distribution. For the following result, we assume that the given stochastic MDP is a mixture of deterministic MDPs, each described by a b-bit random string (see Ng and Jordan [35]). Lemma 5.1. Let stochastic MDPM and policy π be given. DefineRπ(M) to be the expected reward on MDPM. Then the probability that the policy’s expected return on an observed (deterministic) MDP M̂ is much different than the policy’s expected return on the stochastic MDP is bounded by the policy’s bitrate:
PM̂[|R π(M̂)−Rπ(M)| > ] ≤ C + 1
2b 2 − 1 .
The proof is a direct application of Bassily et al. [4, Theorem 8]. This result may be of interest in the offline RL setting, where observed trajectories effectively constitute a deterministic MDP; that policies that use fewer bits will be less likely to overfit to the offline dataset.
Not only do compressed policies generalize to new MDPs, they are also robust to missing observations. Intuitively, a policy that uses zero bits of information will perform identically in the open-loop setting. The following result shows that our model compression objective corresponds to maximizing a lower bound on the expected return of the open-loop policy. Let πopen be the open-loop policy corresponding to the composition of the prior m(zt+1 | zt,at) and the high-level policy πz(at | zt). Let πreactive
be the reactive policy corresponding to the composition of the encoder φ(zt | st) and the high-level policy πz(at | zt). Lemma 5.2. Let encoder φ(zt | st), policy πz(at | zt), prior m(zt | zt−1,at−1), and reward function r(st,at) > 0 be given. Then our model compression objective (Eq. 3) with reward function (1− γ) log r(st,at) is a lower bound on the expected return of the open-loop policy.
Eπopen(τ) [ ∞∑ t=1 γtr(st,at) ] ≥ f ( Eπreactive(τ) [ ∞∑ t=1 γt ( (1− γ) log r(st,at) + logm(zt | zt−1,at−1)− log φ(zt | st) )]) ,
where f(x) = γ1−γ e x γ is a monotone increasing function of x.
Moreover, the policies learned by RPC will receive similar returns in settings where they can and cannot observe observations, a result that may be useful for quantifying the regret incurred when using the learned representation zt as an action space of temporally-extended behaviors. Let Rmax = maxτ R(τ) be the maximum return of any trajectory.
Lemma 5.3. The expected return of the open-loop policy πopen is at most Rmax √ C/2 worse than the expected return of the reactive policy πreactive:
Eπopen [∑
t
γtr(st,at) ] ≥ Eπreactive [∑ t γtr(st,at) ] −Rmax √ C/2.
In summary, our theoretical results suggest that compressed policies generalize well and, by virtue of their open-loop guarantees, may be useful for planning. We emphasize that the theoretical benefits of model compression have been well studied in the supervised learning literature. As our method is likewise performing model compression, we expect that it will inherit a wide range of additional guarantees, such as guarantees about sample complexity.

heading: 6 Experiments

text:
Our experiments have two aims. First, we will demonstrate that RPC achieves better compression than alternative approaches, obtaining a higher reward for the same number of bits. Second, we will study the empirical properties of compressed policies learned by our method, such as their robustness and ability to learn representations suitable for hierarchical RL. We do not intend this section to exhaustively demonstrate every possible benefit from compression; we acknowledge that there are many purported benefits of compression, such as exploration and sample efficiency, which we do not attempt to study here. We include additional experiments in Appendix B.
6.1 Evaluating Compression
feedforward architecture predicted prior augmented reward
RPC (ours) 3 3 3 VIB [21] 3 7 7 VIB+reward [31] 3 7 7 VIB+RNN 7 7 3 Our first experiment studies whether RPC outperforms alternative approaches to compression, which we summarize in the inline table. We compare against a standard VIB [21] and an extension that adds the information cost to the reward, “VIB+reward” [31]. This baseline can be viewed as a special case of RPC where the blue arrows in Fig. 1 are removed. Finally, since the predictive prior in RPC is similar to a recurrent network, we compare against an extension of the VIB [21] that uses an LSTM (“RNN + VIB”). Unlike RPC, this baseline requires training on entire trajectories (rather than individual transitions) and requires backpropagating gradients through time. We evaluate all methods on four tasks from
OpenAI-Gym [5] and two image-based tasks from dm-control [40]. Because of computational constraints, we omit the RNN+VIB baseline on the image-based tasks.
We plot results in Fig. 2. To make the rewards comparable across tasks, we normalize the total return by the median return of the best method. On almost all tasks, RPC achieves higher returns than prior methods for the same bitrate. For example, when learning the Walker task using 0.3 bits per observation, RPC achieves a return of 3,562, the VIB+RNN baseline achieves a return of 1,978 (-44%), and the other VIB baselines achieve a return of around 530 (-85%). We include full results on a wider range of bitrates in Appendix Fig. 10. While, in theory, the VIB+RNN baseline could implement RPC internally, in practice it achieved lower returns, perhaps because of optimization challenges associated with training LSTMs [7]. Even if the VIB+RNN baseline could implement the strategy learned by RPC, RPC is simpler (it does not require training on trajectories) and trains about 25% times faster (it does not require backpropagation through time).

heading: 6.2 Visualizing Compressed Policies

text:
Behavior of compressed policies. To visualize how compression changes behavior, we applied RPC to two simulated driving tasks shown in Fig. 3a (top left), which are based on prior work [29]. In the first task, the agent can pass cars by driving into the lane for oncoming traffic; the second task uses the same simulated environment but restricts the agent to remain in its own lane to study active cruise control in isolation. The rewards for these tasks corresponds to driving to the right as quickly as possible, without colliding with any other vehicles. In the first task, we observe that compressed policies passed fewer cars than uncompressed policies. Intuitively, a passing maneuver requires many bits of information about the relative positions of other cars, so we expect that compressed policies to engage in fewer passing maneuvers. Fig. 3a (bottom) shows that the bitrate of RPC is directly correlated with the number of cars passed. In the second task, we observe that compressed policies leave a larger following distance from the leading car (Fig. 3b (top)). Fig. 3b (bottom) shows that the number of bits used per observation increases when the car is within 15m of another car. Thus, by maintaining a following distance of more than 30m, the compressed policy can avoid these situations where it would have to use more bits. See the project website for videos.3
Representations of compressed policies. A policy learned by compression must trade off between maximizing reward and paying to receive bits of information. Using the HalfCheetah-v2 task, we plot the value of information versus cost of information for many sampled states. See Appendices A.4 and B.5 for details. As shown in Fig. 4 (top), the RPC uses more bits from observations that are more valuable for maximizing future returns. Then, to visualize the learned representation zt, we sample the representation from randomly sampled states, and plot the numbers of bits used for each coordinate. Fig. 4 (bottom) shows that RPC learns sparse representations. Whereas the uncompressed policy uses all coordinates, a policy compressed with bitrate 10 uses only 10/50 coordinates and a policy compressed with bitrate 0.3 uses only 2/50 coordinates.
Intuitively, the ways in which compression changes the behavior and representations of policies should make these policies more robust to disturbances. Our next experiments explicitly test this hypothesis, which would validate the analysis in Sec. 5.2.
3Project site with videos and code: https://ben-eysenbach.github.io/rpc

heading: 6.3 Robustness

text:
The connection between compression and robustness has been well established in the literature (e.g., [16, 47]). Our next set of experiments test the robustness of compressed policies to different types of disturbances: missing observations (i.e., open-loop control), adversarial perturbations to the observations, and perturbations to the dynamics (i.e., robust RL). We emphasize that, because these experiments focus on robustness, the policy is trained and tested in different environments.
Robustness to missing observations and open-loop control. Since compressed policies rely on fewer bits of input from the observations, we expect that they not only will be less sensitive to missing observations, but also will actively modify their behavior to adopt strategies that require fewer bits from the observation. In this experiment, we drop each observation
independently with probability p ∈ [0, 1], where p = 1 corresponds to using a fully open-loop policy. For RPC, we handle missing observations by predicting the representation from the previous time step. Our two main baselines take a policy used by standard RL and learn either a latent-space model or a state-space model. When the observation is missing, these baselines make predictions using the learned model. We also compare against RNN+VIB, the strongest baseline from Fig. 2. When observations are missing, the LSTM’s input for that time step is sampled from the prior. Fig. 5 shows that all methods perform similarly when no observations are dropped, but RPC achieves a higher reward than baselines when a larger fraction of observations are dropped. This experiment shows that more effective compression, as done by RPC, yields more robust policies. This robustness may prove useful in real-world environments where sensor measurements are missing or corrupted.
Adversarial Robustness. Compressed policies extract fewer bits from each observation, so we expect that compressed policies will be more robust to adversarial perturbations to the observation. While prior work has proposed purpose-designed methods for achieving robustness [33, 41], here we investigate whether compression is simple yet effective means for achieving a modicum of robustness. We do not claim that compression is the best method for learning robust policies.
We first study adversarial perturbations to the dynamics on the Ant-v2 environment. Given a policy π(at | st) and the current state st, the adversary aims to apply a small perturbation to that state to make the policy perform as poorly as possible. We implement the adversary using projected gradient descent [32]; see Appendix B for details. Fig. 6 (left) shows the expected re-
turn as we increase the magnitude of the attack. The compressed policy is more resilient to larger attacks than the uncompressed policy. Our next experiment looks at perturbations to observations. Unlike the previous experiment, we let the adversary perturb every step in an episode; see Appendix B for full details. Fig. 6 (right) shows that policies that use fewer bits achieve higher returns in this adversarial setting. When the Ant-v2 agent uses too many bits (3 or more), the adversarial perturbations to the dynamics flip the agent over, whereas agents that use fewer bits remain upright.
Robust RL. Our final set of experiments look at higher-level perturbations to the dynamics, as are typically studied in the robust RL community [33, 41]. Using the same Ant-v2 environment as before, we (1) increase the mass of each body element by a fixed multiplier, or (2) decrease the friction of each body geometry by a fixed multiplier. These experiments test whether the learned policies are robust to more massive
robots or more “slippery” settings. Fig. 7 shows that compressed policies generalize to larger masses and smaller frictions more effectively than an uncompressed policy. Comparing the policies learned
by RPC with a bit rate of 0.3 bits versus 3.0 bits, we observe that the bit rate effectively balances performance versus robustness. See Appendix Fig. 11 for a larger version of this plot with error bars.

heading: 6.4 Hierarchical RL using the Learned Representation

text:
RPC learns a representation of temporally-extended action sequences (see Sec. 5.1). We hypothesize that these representations can accelerate the learning of new tasks in a hierarchical setting. Our goal is not to propose a complete hierarchical RL system, but rather evaluate whether these action representations are suitable for high-level control. During training, we apply RPC to a goal-conditioned object pushing task, shown in Fig. 8 (top-left). The initial position of the object and the goal position are randomized, so we expect that different representations zt will correspond to high-level behaviors of moving the end effector and object to different positions. At test-time, the agent is presented with a new task. The agent will attempt to solve the task by commanding one or two behaviors zt. See Appendix B for details and pseudocode.
We compare the action representations learned by RPC to three baselines. The first baseline is “Action Repeat”, which constantly outputs the same action, effectively allowing the agent to command desired poses. To test whether RPC has learned a prior over useful action sequences, we use a variant of RPC (“RPC (randomly initialized)” ) where the policy and model are randomly initialized. Finally, “RL from scratch” applies a state-of-the-art off-policy RL algorithm (SAC [17]) to the task. Taken together, these baselines allow us to study whether RPC not only learns behaviors that do more than move to particular poses, but also learns behaviors for obstacle avoidance and object manipulation. We apply all methods to four tasks and present results in Fig. 8. First, as a sanity check, we apply all methods to the training task, finding that RPC quickly finds a single zt that solves the task. On the remaining three tasks, we likewise observe that the representations learned by RPC allow for much faster learning than the baselines. The final task, “pusher wall” requires chaining together multiple representations zt to solve. While the “RL from scratch” baseline eventually matches and then surpasses the performance of RPC, RPC accelerates learning in the low-data regime.

heading: 7 Conclusion

text:
In this paper, we presented a method for learning robust and predictable policies. Our objective differs from prior work by compressing sequences of observations, resulting in a method that jointly trains a policy and a model to be self-consistent. Not only does our approach achieve better compression than prior methods, it also learns policies that are more robust (Fig. 6). We also demonstrate that our method learns representations that are suitable for use in hierarchical RL.
Our work suggests a few areas for future work. Ideas based on RPC may be useful for model-based RL, as training the policy to be self-consistent with the model may help prevent model exploitation. Similarly, because the skills learned by RPC are optimized to be predictable, we expect that they will perform well when used for open-loop planning. As compression is closely related to the energy required to implement a function on an ideal physical system [37], RPC might be useful to building energy-efficient control policies.
Limitations. The main limitation of this work is that policies that use few bits will often receive lower reward on the training tasks. Also, for the purpose of exploration, the most informative states may be those that are hardest to compress. While the first limitation is likely irreconcilable, the second might be lifted by maximizing information collected during exploration but minimizing information for policy optimization.
Acknowledgments. We thank Dibya Ghosh and the members of the Salakhutdinov lab for feedback throughout the project. We thank Oscar Ramirez for help setting up the image-based experiments and thank Rico Jonschkowski for introductions to potential collaborators. We thank Ryan Julian, Vincent Vanhoucke, and anonymous reviewers for feedback on the paper, and thank Abhishek Gupta for discussions during the early stages of this project.
This material is supported by the Fannie and John Hertz Foundation and the NSF GRFP (DGE1745016).

title: Robust Predictable Control