summaries: The paper investigate how optimal recurrent neural networks (RNNs) are at storing past information such that it is useful for predicting the future. The authors estimated optimality in terms of mutual information between the past and the future. If the RNN was able to retain MI between the past and the future, it then has kept optimal information from the past for predicting the future. The experiments suggest that RNNs are not optimal in terms of prediction of the future. It also suggest that this is due to the maximum likelihood training objective. This manuscript shows that good ability to compress past information in RNNs help them to predict the future, and that improving upon this ability leads to more useful RNNs. The manuscript first adapts modern mutual-information estimators to mini-batch settings in order to measure the information that an RNN has on the past. It then considers stochastic training, adding Gaussian noise to the hidden states during the training of the RNNs to limit past information. A significant section is dedicated to an empirical study that shows that classically-train MLE RNNs lead to internal representations with a suboptimal mutual-information to the past and the future. For LSTM and GRU architecture, stochastic training actually significantly helps. Experiments on applications such as synthetizing hand-drawn sketches suggest that stochastic training leads to more useful RNNs. This paper certainly poses an interesting question: How well do RNNs compress the past while retaining relevant information about the future. In order to quantitatively answer this question, the authors suggest to look at (the optimal solutions of) the Information Bottleneck Lagrangian (IBL). The investigated RNNs need to solve the task of next-step prediction, which can be used to evaluate the IBL. In the paper, the (deterministic) hidden state h is transformed through simple additive Gaussian noise into a stochastic representation which then is utilized to compute the IBL. In general the IBL is not tractable and hence the paper uses approximate computations.
abstractText: Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work, we investigate if the same can be said of artificial neurons in recurrent neural networks (RNNs) trained with maximum likelihood. In experiments on two datasets, restorative Brownian motion and a hand-drawn sketch dataset, we find that RNNs are sub-optimal in the information plane. Instead of optimally compressing past information, they extract additional information that is not relevant for predicting the future. We show how constraining past information by injecting noise into the hidden state can improve the ability of RNNs to extract predictive information for both maximum likelihood and contrastive loss training.
references.authors: D Barber FV Agakov
references.title: The IM algorithm: A variational approach to information maximization
references.venue: In NIPS,
references.authors: William Bialek Ilya Nemenman Naftali Tishby
references.title: Predictability, complexity, and learning
references.venue: Neural computation,
references.authors: Gal Chechik Amir Globerson Naftali Tishby Yair Weiss
references.title: Information bottleneck for gaussian variables
references.venue: Journal of Machine Learning Research,
references.authors: Kyunghyun Cho Bart van Merrienboer Çaglar Gülçehre Fethi Bougares Holger Schwenk Yoshua Bengio
references.title: Learning phrase representations using RNN encoder-decoder for statistical machine
references.venue: translation. CoRR,
references.authors: Junyoung Chung Kyle Kastner Laurent Dinh Kratarth Goel Aaron C Courville Yoshua Bengio
references.title: A recurrent latent variable model for sequential data
references.venue: In Advances in neural information processing systems,
references.authors: Thomas M Cover Joy A Thomas
references.title: Elements of information theory
references.authors: Xavier Glorot Yoshua Bengio
references.title: Understanding the difficulty of training deep feedforward neural networks
references.venue: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: In Advances in Neural Information Processing Systems
references.authors: David Ha Douglas Eck
references.title: A neural representation of sketch drawings
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
references.venue: In The IEEE International Conference on Computer Vision (ICCV),
references.authors: Sepp Hochreiter Jürgen Schmidhuber
references.title: Long short-term memory
references.venue: Neural Comput.,
references.authors: Diederik P. Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In 3rd International Conference on Learning Representations,
references.authors: David McAllester Karl Stratos
references.title: Formal limitations on the measurement of mutual information, 2019
references.venue: URL https://openreview.net/forum?id=BkedwoC5t7
references.authors: Simon F. Nørrelykke Henrik Flyvbjerg
references.title: Harmonic oscillator in heat bath: Exact simulation of time-lapse-recorded data and exact analytical benchmark statistics
references.venue: Phys. Rev. E,
references.authors: Aaron van den Oord Yazhe Li Oriol Vinyals
references.title: Representation learning with contrastive predictive coding
references.venue: arXiv preprint arXiv:1807.03748,
references.authors: Stephanie E Palmer Olivier Marre Michael J Berry William Bialek
references.title: Predictive information in a sensory population
references.venue: Proceedings of the National Academy of Sciences,
references.authors: Ben Poole Sherjil Ozair Aaron Van Den Oord Alex Alemi George Tucker
references.title: On variational bounds of mutual information
references.venue: In International Conference on Machine Learning,
references.authors: Ning Qian
references.title: On the momentum term in gradient descent learning algorithms
references.venue: Neural Networks,
references.authors: Nitish Srivastava Geoffrey E. Hinton Alex Krizhevsky Ilya Sutskever Ruslan Salakhutdinov
references.title: Dropout: a simple way to prevent neural networks from overfitting
references.venue: Journal of Machine Learning Research,
references.authors: Naftali Tishby Fernando C.N. Pereira William Bialek
references.title: The information bottleneck method
references.venue: CoRR, physics/0004057,
references.authors: Gašper Tkačik William Bialek
references.title: Information processing in living systems
references.venue: Annual Review of Condensed Matter Physics,
references.authors: Aaron van den Oord Yazhe Li Oriol Vinyals
references.title: Representation learning with contrastive predictive coding
references.venue: arXiv preprint arXiv:1807.03748,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In Advances in neural information processing systems,
sections.heading: 1 INTRODUCTION
sections.text: Remembering past events is a critical component of predicting the future and acting in the world. An information-theoretic quantification of how much observing the past can help in predicting the future is given by the predictive information (Bialek et al., 2001). The predictive information is the mutual information (MI) between a finite set of observations (the past of a sequence) and an infinite number of additional draws from the same process (the future of a sequence). As a mutual information, the predictive information gives us a reparameterization independent, symmetric, interpretable measure of the co-dependence of two random variables. More colloquially, the mutual information tells us how many bits we can predict of the future given our observations of the past.1 Asymptotically, a vanishing fraction of the information in the past is relevant to the future (Bialek et al., 2001), thus systems which excel at prediction need not memorize the entire past of a sequence.
  Intriguingly, certain biological neurons extract representations that efficiently capture the predictive information in sequential stimuli (Palmer et al., 2015; Tkačik & Bialek, 2016). In Palmer et al. (2015), spiking responses of neurons in salamander retina had near optimal mutual information with the future states of sequential stimuli they were exposed to, while compressing the past as much as possible.
  Do our artificial neural networks perform similarly? In this work, we aim to answer this question by measuring artificial recurrent neural networks’ (RNNs) capacity to compress the past while retaining relevant information about the future.
  Our contributions are as follows:
  • We demonstrate that RNNs are suboptimal at extracting predictive information on the tractable sequential stimuli used in Palmer et al. (2015).
  • We thoroughly validate the accuracy of our mutual information estimates on RNNs and optimal models.
  • We repeat our analysis on the real-world Aaron Koblin Sheep sketch dataset (Ha & Eck, 2017), demonstrating that deterministic RNNs continue to perform suboptimally.
  • We demonstrate that on small datasets, models trained to compress the past have improved sample quality over naively-trained RNNs.
  1 Because of its symmetry, this is equivalent to the number of bits we can reconstruct of the past given observations of the future.
sections.heading: 2 BACKGROUND AND METHODS
sections.text: We begin by providing additional background on predictive information, mutual information estimators, stochastic RNNs, and Gaussian Information Bottleneck. These tools are neccessary for accurately evaluating the question of whether RNNs are optimal in the information plane, as we require knowledge of the optimal frontier, and accurate estimates of mutual information for complex RNN models.
sections.heading: 2.1 PREDICTIVE INFORMATION
sections.text: Imagine an infinite sequence of data: Xt. The predictive information (Bialek et al., 2001) in the sequence is the mutual information between some finite number of observations of the past (T ) and the infinite future of the sequence:
  Ipred(T ) = I(Xpast;Xfuture) = I({Xt−T+1, . . . , Xt}; {Xt+1, . . . }). (1)
  For a process for which the dynamics are not varying in time, this will be independent of the particular time t chosen to be the present. More specifically, the predictive information is an expected log ratio between the likelihood of observing a future given the past, versus observing that future in expectation over all possible pasts:
  Ipred ≡ E [ log
  p(xfuture|xpast) p(xfuture)
  ] = E [ log
  p(xfuture|xpast) Ex′past [ p(xfuture|x′past) ]] . (2) A sequential model such as an RNN provides a stochastic representation of the entire past of the sequence Z ∼ p(z|xpast). For any such representation, we can measure how much information it retains about the past, a.k.a. the past information: Ipast = I(Z;Xpast), and how informative it is about the future, a.k.a. the future information: Ifuture = I(Z;Xfuture). Because the representation depends only on the past, our three random variables satisfy the Markov relations: Z ← Xpast ↔ Xfuture and the Data Processing Inequality (Cover & Thomas, 2012) ensures that the information we have about the future is always less than or equal to both the true predictive information of the sequence (Ifuture ≤ Ipred) as well as the information we retain about the past (Ifuture ≤ Ipast). For any particular sequence, there will be a frontier of solutions that optimally tradeoff between Ipast and Ifuture. A common method for tracing out this frontier is through the Information Bottleneck (Tishby et al., 2000) Lagrangian:
  min p(z|xpast)
  I(Z;Xpast)− βI(Z;Xfuture), (3)
  where the parameter β controls the tradeoff. An efficient representation of the past is one that lies on this optimal frontier, or equivalently is a solution to Eqn. 3 for a particular choice of β. For simple problems, where the sequence is jointly Gaussian, we will see that the optimal frontier can be identified analytically.
sections.heading: 2.2 MUTUAL INFORMATION ESTIMATORS
sections.text: In order to measure whether a representation is efficient, we need a way to measure its past and future informations. While mutual information estimation is difficult in general (McAllester & Stratos, 2019), recent progress has been made on a wide range of variational bounds on mutual information (Poole et al., 2019). While these provide bounds and not exact estimates of mutual information, they allow us to compare mutual information quantities in continuous spaces across models. There are two broad families of estimators: variational lower bounds powered by a tractable generative model, or contrastive lower bounds powered by an unnormalized critic.
  The former class of lower bounds, first presented in Barber & Agakov (2003), are powered by a variational generative model:
  Ifuture = E [ log
  p(xfuture|z) p(xfuture)
  ] ≥ H(xfuture) + E [log q(xfuture|z)] . (4)
  A generative model provides a demonstration that there exists at least some information between the representation z and the future of the sequence. For our purposes, H(xfuture), the entropy of the
  future of the sequence is a constant, determined by the dynamics of the sequence itself and outside our control. For tractable problems, such as the toy problem we investigate below, this value is known. For real datasets, this value is not known so that we cannot produce reliable estimates of the mutual information. It does, however, still provide reliable gradients of a lower bound. One example of such a generative model is the loss used to train the RNN to begin with.
  Contrastive lower bounds can be used to estimate Ifuture for datasets where building a tractable generative model of the future is challenging. InfoNCE style lower bounds (van den Oord et al., 2018; Poole et al., 2019) only require access to samples from both the joint distribution and the individual marginals:
  I(X;Z) ≥ INCE(X;Z) , EpK(x,z)
  [ 1
  K K∑ i=1 log ef(xi,zi) 1 K ∑K j=1 e f(xj ,zi)
  ] . (5)
  Here f(xj , zi) is a trained critic that plays a role similar to a discriminator in a Generative Adversarial Network (Goodfellow et al., 2014). It scores pairs, attempting to determine if an (x, z) pair came from the joint (p(x, z)) or the factorized marginal distributions (p(x)p(z)).
  When forming estimates of Ipast, we can leverage additional knowledge about the known encoding distribution from the stochastic RNN p(z|xpast) to form tractable upper and lower bounds without having to learn an additional critic (Poole et al., 2019):
  E
  [ 1
  K K∑ i=1 log p(zi|xipast) 1 K ∑ j p(z i|xjpast)
  ] ≤ I(Z;Xpast) ≤ E [ 1
  K K∑ i=1 log p(zi|xipast) 1 K−1 ∑ j 6=i p(z i|xjpast)
  ] . (6)
  We refer to these bounds as minibatch upper and lower bounds as they are computed using minibatches of size K from the dataset. As the minibatch size K increases, the upper and lower bounds can become tight. When logK I(Z;Xpast) the lower bound saturates at logK and the upper bound can be loose, thus we require using large batch sizes to form accurate estimates of Ipast.
sections.heading: 2.3 STOCHASTIC RNNS
sections.text: Deterministic RNNs can theoretically encode infinite information about the past in their hidden states (up to floating point precision). To limit past information, we devise a simple stochastic RNN. Given the deterministic hidden state ht, we output a stochastic variable zt by adding i.i.d. Gaussian noise to the hidden state before reading out the outputs: zt ∼ N (ht, σ2). These stochastic outputs are then used to predict the future state: x̂t+1 ∼ N (gdecoder(zt), σ2o), as illustrated in Figure 1. With bounded activation functions on the hidden state ht, we can use σ2 to upper bound the information stored about the past in the stochastic latent zt. This choice of stochastic recurrent model yields a tractable conditional distribution p(zt|x≤t) ∼ N (ht, σ2), which we can use in Eq. 6 to form tractable upper and lower bounds on the past information. We will consider two different settings for our stochastic RNNs: (1) where the RNNs are trained deterministically and the noise on the hidden state is added only at evaluation time, and (2) where the RNNs are trained with noise, and evaluated with noise.
sections.heading: 2.4 GAUSSIAN INFORMATION BOTTLENECK
sections.text: To evaluate optimality of RNNs, we first focus on the tractable sequential stimuli of a simple Brownian Harmonic Oscillator as used in (Palmer et al., 2015). A crucial property of this dataset is that we can analytically calculate the optimal trade-off between past and future information as it is an instance of the Gaussian Information Bottleneck (Chechik et al., 2005).
  Consider jointly multivariate Gaussian random variables X ∈ RDX and Y ∈ RDY , with covariance ΣX and ΣY and cross-covariance ΣXY . The solution to the Information Bottleneck objective:
  min T I(X;T )− βI(Y ;T ), (7)
  is given by a linear transformation T = AX + ε with ε ∼ N (0,Σε). The projection matrix A projects along the lowest eigenvectors of ΣX|Y Σ −1 X , where the trade-off parameter β decides how many of the eigenvectors participate. Further details can be found in Appendix A.1.
sections.heading: 3 RESULTS
sections.heading: 3.1 BROWNIAN HARMONIC OSCILLATOR
sections.text: We begin by considering the Brownian harmonic oscillator (BHO) dataset, a form of restorative Brownian motion, as in Palmer et al. (2015). This system has the benefit of having fully tractable
  predictive information. The dynamics are given by:
  xt+∆t = xt + vt∆t, vt+∆t = [1− Γ∆t]vt − ω2xt∆t+ ξt √ D∆t.
  (8)
  where ξt is a standard Gaussian random variable. Additional details can be found in Appendix A.2. Examples of the trajectories for this system can be seen in Figure 2a.
  Given its analytical tractability we can explicitly compare the estimated RNN performance against optimal performance. We compared three major variants of RNNs, including fully-connected RNNs, gated recurrent units (GRU, Cho et al. (2014)), and LSTMs (Hochreiter & Schmidhuber, 1997). Each network had 32 hidden units and tanh activations. Full training details are in Appendix A.2.
  Figure 2 contains the results for both stochastic RNNs and deterministic RNNs with noise added only at evaluation time. By varying the strength of the noise, networks trace out a trajectory on the information plane. We find that networks trained with noise are close to the optimal frontier (black), in terms of optimally extracting information about the past that is useful for predicting the future. While injecting noise at evaluation time produces networks with compressed representations, these deterministically trained networks perform worse than their stochastically trained counterparts.
  Training with noise injection can make the networks more efficiently capture the predictive information than post-hoc noise injection. At the same noise level (the color coding in Figure 2) stochastically trained RNNs have both higher I(z;xpast) and higher I(z;xfuture). By limiting the capacity of the models during training, they were able to extract more efficient representations. For this task, we find that more complex RNN variants such as LSTMs are less efficient at encoding predictive information.
  Dropout (Srivastava et al., 2014) is a potential alternative way to eliminate information. In Appendix A.5, we demonstrate that RNNs trained with dropout extract less information than the ones trained without dropout, and that our simple noise injection technique can find equivalent models. To access whether the observed sub-optimality was a consequence of our choice of maximum likelihood (MLE) as a training objective, in Appendix A.4 we demonstrate the effect of using contrastive predictive coding (CPC) (Oord et al., 2018) as the training objective. We find that models trained with CPC loss perform similarly to models trained with MLE, as measured on the information plane. However, this may be due to the BHO dataset having Markovian dynamics, and optimizing for onestep-ahead prediction with MLE is sufficient to maximize mutual information with the future of the sequence.
sections.heading: 3.2 ACCURACY OF MUTUAL INFORMATION ESTIMATES
sections.text: Our claim that RNNs are suboptimal in capturing predictive information hinges on the quality of our MI estimates. Are the RNNs truly suboptimal or is it just that our MI estimates are inaccurate? Here we provide several experiments further validating the accuracy of our MI estimates.
  Comparison among estimators. There are many different mutual information estimators. In Figure 3, we compare various mutual information lower bounds with learned critics: InfoNCE, NWJ and JS, as summarized in Poole et al. (2019). NWJ and JS show higher variance and worse bias than InfoNCE. The second panel of Figure 3 demonstrates that InfoNCE outperforms a variational Barber-Agakov style variational lower bound at measuring the future information. Therefore, we adopted InfoNCE as the critic based estimator for the future information in the previous section.
  For the past information, we could generate both tractable upper and lower bounds, given our tractable likelihood, p(zt|x≤t ∼ N (ht, σ2). In the third panel of Figure 3 we demonstrate that these bounds become tight as the sample size increases. However they require a large number of samples before they converge. Fundamentally, the lower bound itself is upper-bounded by the log of the number of samples used.
  Estimator training with finite dataset. Training the learned critic on finite datasets for a large number of iterations resulted in problematic memorization and overestimates of MI. To counteract the overfitting, we performed early stopping using the MI estimate with the learned critic on a validation set. Unlike the training MI, this is a valid lower bound on the true MI. We then report estimates of mutual information using the learned critic on an independent test set, as Figure 4.
  Analytical Agreement As a final and telling justification of the efficiency of our estimators, Figure 5 demonstrates that our estimators accurately estimate the mutual information in our region of interest,
  for analytically derived optimal projections. Background for the Gaussian Information Bottleneck is in Section 2.4 and details of the calculation can be found in Appendix A.1.
sections.heading: 3.3 VECTOR DRAWING DATASET
sections.text: In order to assess whether the sub-optimality of RNNs was an artifact of the BHO dataset, we performed additional experimentson a real world sequential dataset. We used the Aaron Koblin Sheep Sketch Dataset2. Full experimental details are in Appendix A.3. We adopted the SketchRNN architecture and online data augmentation procedure used in Ha & Eck (2017).
  Figure 6 (Left) shows the estimates on the information plane for the trained networks. Again, the networks that were trained explicitly with noise dominate on the information plane. For this real world task we no longer know the optimal information tradeoff, but still see that the deterministically trained networks evaluated with noise are demonstrably suboptimal compared to some simple stochastic networks trained with noise injection.
  Besides demonstrating sub-optimality in capturing predictive information on the information plane, in Figure 6 (Right), we demonstrate that the deterministic RNNs are wasteful. Networks with higher levels of compression (marked by higher noise levels) were able to obtain similar average test set performance with noticeably lower variance than the deterministic networks. They also show a smaller generalization gap. By explicitly training the networks to operate within some constrained information budget reduced variance between runs without substantially reducing test set performance.
  In Figure 7 we show some generated completions from the models trained on the full dataset with varying degrees of noise. The models trained with the highest noise levels show a noticeable degredation in sample quality.
sections.heading: 3.4 SMALL DATASETS
sections.text: We expect the benefits of training compressed representations to be most marked on limited datasets where the compression can prevent the network from memorizing too many spurious correlations.
  To investigate, we repeated the experiments of the previous section with limited dataset with only 100 examples. Figure 8 (Upper Left) shows the corresponding information plane points for stochastically trained RNNs with various noise levels. Notice that at about 4 nats of past information, the networks future information essentially saturates. While it seems as though the networks don’t suffer even when asked to learn richer representations, this is largely an artifact of our training procedure which included early stopping. As can be seen in Figure 8 (Lower Left), all of our networks overfit in terms of evaluation loss, but the onset of overfitting was strongly controlled by the degree of compression.
  2Available from https://github.com/hardmaru/sketch-rnn-datasets/tree/master/ aaron_sheep
  Most noticeably, in the limited data regime, compressed representations lead to improved sample quality, as seen in Figure 8 (Right). Models with intermediately-sized compressed representations show the best generated samples while retaining a good amount of diversity. Models with either too little or too much past information tend to produce nonsensical generations.
sections.heading: 4 CONCLUSION AND DISCUSSION
sections.text: In this work, we have demonstrated how analyzing recurrent neural networks in terms of predictive information can be a useful tool in probing and understanding behavior. We find that RNNs trained with maximum likelihood are sub-optimal on the information plane, extracting more information about the past than is required to predict the future. By analyzing different training objectives and noise injection approaches in the information plane, we can better understood the tradeoffs made by different models, and identify models that are closer to the optimality demonstrated by biological neurons (Palmer et al., 2015).
  While the simple strategy of adding noise to a bounded hidden state can be used to constrain information, setting the amount of noise and identifying where one should be on the information plane remains an open problem. Additionally, studying the impact of other architecture choices, such as stochastic latent variables in variational RNNs (Chung et al., 2015), or attention-based Transformers (Vaswani et al., 2017) in the information plane could yield insights into their better performance on several tasks.
sections.heading: A APPENDIX
sections.heading: A.1 GAUSSIAN INFORMATION BOTTLENECKS
sections.text: Consider jointly multivariate Gaussian random variables X ∈ RDX and Y ∈ RDY , with covariance ΣX and ΣY and cross-covariance ΣXY . The solution to the Information Bottleneck objective:
  min T I(X;T )− βI(Y ;T ), (9)
  is given by a linear transformation T = AX + ε with ε ∼ N (0,Σε). The projection matrix A projects along the lowest eigenvectors λi(i ∈ [1, DX ]) of ΣX|Y Σ−1X , where the trade-off parameter β decides how many of the eigenvectors participate, vTi (i ∈ [1, DX ]). The projection matrix A could be analytically derived as
  A = [α1v T 1 , α2v T 2 , . . . , αDXv T DX ], Σε = I (10)
  where the projection coefficients α2i = max( β(1−λi)−1 λiri , 0), ri = v T i ΣXvi, with proof in Appendix A.1.1.
  Given the optimally projected states T , the optimal frontier (black curve in Figure 5) is:
  I(T ;Y ) = I(T ;X)− nI 2 log( nI∏ i=1 (1− λi) 1 nI + e 2I(T ;X) nI nI∏ i=1 λ 1 nI i ), cnI ≤ I(T ;X) ≤ cnI+1 (11)
  where nI is the cutoff number indicating the number of smallest eigenvalues being used. The critical points cnI , changing from using nI = N eigenvalues to N + 1 eigenvalues, can be derived given the concave and C1 smoothness property for the optimal frontier, with proof in Appendix A.1.2:
  cnI = 1
  2 N∑ i=1 log λN+1 λi 1− λi 1− λN+1
  (12)
sections.heading: A.1.1 PROOF OF OPTIMAL PROJECTION
sections.text: By Theorem 3.1 of Chechik et al. (2005), the projection matrix for optimal projection is given by
  A =  [ α1v T 1 ,0,0, . . . ,0 ] , 0 ≤ β ≤ β1[ α1v T 1 , α2v T 2 ,0, . . . ,0 ] , β1 ≤ β ≤ β2[ α1v T 1 , α2v T 2 , α2v T 2 , . . . ,0 ] , β2 ≤ β ≤ β3
  ...  (13) where vTi (i ∈ [1, DX ]) are left eigenvectors of ΣX|Y Σ −1 X sorted in ascending order by the eigenvalues λi(i ∈ [1, DX ]); βi = 11−λi are critical values for trade-off parameter β; and the projection coefficients are α2i = β(1−λi)−1 λiri
  , ri = vTi ΣXvi. In practice, noticing that β∗(1−λi)−1 < 0 when β < βi, we simplify Equation (13) as A = [α1vT1 , α2v T 2 , . . . , αDXv T DX ] with α2i = max( β(1−λi)−1 λiri , 0).
sections.heading: A.1.2 PROOF OF CRITICAL POINTS ON OPTIMAL FRONTIER
sections.text: By Eq.15 of Chechik et al. (2005)
  I(T ;Y ) = I(T ;X)− nI 2 log( nI∏ i=1 (1− λi) 1 nI + e 2I(T ;X) nI nI∏ i=1 λ 1 nI i ) (14)
  where nI is the cutoff on the number of eigenvalues used to compute the bound segment, with eigenvalues sorted in ascending order.
  In order to calculate the changing point, where one switching from choosing nI = N to N + 1, by C1 smoothness conditions:
  dInI=N (T ;Y )
  dI(T ;X) =
  dInI=N+1(T ;Y )
  dI(T ;X) (15)
  LHS is
  1− dInI=N (T ;Y ) dI(T ;X) =
  N∏ i=1 (λi) 1 N e 2I(T ;X) N
  N∏ i=1 (1− λi) 1 N + e 2I(T ;X) nI N∏ i=1 λ 1 N i
  (16)
  = e
  2I(T ;X) N
  e 2I(T ;X) N + N∏ i=1 ( 1−λiλi ) 1 N
  (17)
  Thus, Equation (15) could be rewritten as
  e 2I(T ;X) N
  e 2I(T ;X) N + N∏ i=1 ( 1−λiλi ) 1 N = e
  2I(T ;X) N+1
  e 2I(T ;X) N+1 + N+1∏ i=1 ( 1−λiλi ) 1 N+1
  (18)
  Rewrite RHS of above equation, and noticing 1n(n+1) = 1 n − 1 n+1
  e 2I(T ;X) N+1
  e 2I(T ;X) N+1 + N+1∏ i=1 ( 1−λiλi ) 1 N+1 = e
  2I(T ;X) N
  e 2I(T ;X) N + e 2I(T ;X) N(N+1) N+1∏ i=1 ( 1−λiλi ) 1 N+1
  (19)
  The term in lower right corner could be written as
  N+1∏ i=1 ( 1− λi λi ) 1 N+1 = N∏ i=1 ( 1− λi λi ) 1 N+1 ( 1− λN+1 λN+1 ) 1 N+1 (20)
  = ( N∏ i=1 ( 1− λi λi ) 1 N N∏ i=1 ( λi 1− λi ) 1 N(N+1) ) ( 1− λN+1 λN+1 ) N (N+1)N (21)
  = ( N∏ i=1 ( 1− λi λi ) 1 N )( N∏ i=1 ( λi(1− λN+1) (1− λi)λN+1 ) 1 N(N+1) ) (22)
  Putting RHS together
  e 2I(T ;X) N
  e 2I(T ;X) N + N∏ i=1 ( 1−λiλi ) 1 N = e
  2I(T ;X) N
  e 2I(T ;X) N + e 2I(T ;X) N(N+1) ( N∏ i=1 ( 1−λiλi ) 1 N )( N∏ i=1 ( λi(1−λN+1) (1−λi)λN+1 ) 1 N(N+1) ) (23)
  To let LHS = RHS, one trivial solution is
  1 = e 2I(T ;X) N(N+1) ( N∏ i=1 ( λi(1− λN+1) (1− λi)λN+1 ) 1 N(N+1) ) (24)
  Taking log and cancelling out multiplicative factors, one get the critic point to change from ni = N to ni = N + 1 happens at
  I(T ;X) = 1
  2 N∑ i=1 log λN+1 λi 1− λi 1− λN+1
  (25)
  The original result written in Chechik et al. (2005) missing a factor of 12 .
sections.heading: A.1.3 OPTIMAL PROJECTION
sections.text: The optimal frontier is generated by joining segments described by Equation (11), as illustrated in Figure 9.
sections.heading: A.2 DETAILS FOR BROWNIAN HARMONIC OSCILLATOR
sections.text: To generate the sample trajectories, we set the undamped angular velocity ω = 1.5 × 2π(rad), damping coefficient Γ = 20.0, and the dynamical range of external forces D = 1000.0, with integration time step-size ∆t = 0.01667. The stationary distribution of Equation (8) is analytically derived in Nørrelykke & Flyvbjerg (2011).
  We train RNNs with infinite number of training samples, which are generated online and divided into batches of 32 sequences. RNNs, including fully connected RNN, GRU and LSTM, are all with 32 hidden units and tanh activation. They are trained with momentum optimizer (Qian, 1999) for 20000 steps, with momentum = 0.9 and gradient norm being clipped at 5.0. Learning rate for training is exponentially decayed in a stair-case fashion, with initial learning rate 10−4, decay rate 0.9 and decay steps 2000.
  The mutual information estimators, with learned critics, are trained for 200000 steps with Adam optimizer (Kingma & Ba, 2015) at a flat learning rate of 10−3. The training batch size is 256, and the validation and evaluation batch sizes are 2048. We use early stopping to deal with overfitting. The training is stopped when the estimation on validation set does not improve for 10000 steps, or when it drops by 3.0 from its highest level, whichever comes first. We use separable critics (Poole et al., 2019) for training the estimators. Each of the critics is a three-layer MLP, with [256, 256, 32] hidden units and [ReLU, ReLU, None] activations. The weights for each layer are initialized with Glorot uniform initializer (Glorot & Bengio, 2010), and the biases are with He normal initializer (He et al., 2015). For the minibatch upper and lower bounds, they are estimated on batches of 4096 sequences.
  To train the critics, we feed 100-step BHO sequences into trained RNN to get RNN hidden states and conditional distribution parameters. From each sequence, we use last 36 steps for the inputs to the estimators, where first 18 steps as xpast[t], t = [1, 2, . . . 18], and the other 18 steps as xfuture[t], t = [19, 20, . . . 36]. The hidde state z18 is extracted at the last time step of xpast.
sections.heading: A.3 TRAINING DETAILS FOR VECTOR DRAWING DATASET
sections.text: We train decoder-only SketchRNN (Ha & Eck, 2017) on Aaron Koblin Sheep Dataset, as provided in https://github.com/hardmaru/sketch-rnn-datasets/tree/master/ aaron_sheep. The SketchRNN uses LSTM as its RNN cell, with 512 hidden units.
  For RNN training, We adopt the identical hyper-parameters as in https://github.com/ tensorflow/magenta/blob/master/magenta/models/sketch_rnn/model.py, except that we turn off the recurrent drop-out, since drop-out masks out informations and will interfere with noise injection.
  For mutual information estimations, we use the identical hyper-parameters as descibed in Appendix A.2, except that: the evaluation batch size for critic based estimator, InfoNCE, is set to be 4096, and 16384 for minibatch bounds; early stopping criteria are changed to that either the estimation does not improve for 20000 steps or drops by 10.0 from its highest level, whichever comes first.
  Due to the limitation of the sequence length of Aaron’s Sheep, we use the samples with at least 36 steps long. The xpast and xfuture are split at the middle of the sequences, and each with 18 steps.
  Due to the limitation of the dataset size of Aaron’s Sheep, we augment the dataset with randomly scale the stroke by a factor sampled from N (0, 0.15) for each sequence to generate a large dataset. Figure 4 (Right) shows that the augmentation helps in training the estimator.
sections.heading: A.4 EFFECT OF CHANGING TRAINING OBJECTIVES: MAXIMUM LIKELIHOOD AND CONTRASTIVE LOSS TRAINING
sections.text: To access whether the observed suboptimality in the information plane was due to the maximum likelihood (MLE) objective itself, we additionally trained constrastive predictive coding (CPC) models (Oord et al., 2018). We used the identical model architecture as described in the Section 2.3. The CPC loss lower bounds the mutual information between the current time step and K steps into the future. For our experiments on the Brownian harmonic oscillator (BHO), we look into K = 30 steps future, and use a linear readout from the hidden states to a time-independent embedding of the
  inputs. As shown in Figure 10, we found that models trained with CPC loss had similar frontiers as those trained with MLE. The loss function does not appear to have a substantive effect.
sections.heading: A.5 EFFECT OF DROPOUT
sections.text: Dropout (Srivastava et al., 2014) is a common method applied on neural network training to prevent overfitting. We trained fully connected RNNs and LSTMs with different levels of dropout probability. As shown in Figure 11, we find that RNNs trained with dropout extract less information than the ones without it, but the information frontier of the models does not change, when we sweep dropout rate and additive noise. In other words, we can find models that are equivalent to dropout models by simply adding Gaussian noise to the output at training time.