DEEP INFORMATION PROPAGATION
We study the behavior of untrained neural networks whose weights and biases are randomly distributed using mean field theory. We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks. Our main practical result is to show that random networks may be trained precisely when information can travel through them. Thus, the depth scales that we identify provide bounds on how deep a network may be trained for a specific choice of hyperparameters. As a corollary to this, we argue that in networks at the edge of chaos, one of these depth scales diverges. Thus arbitrarily deep networks may be trained only sufficiently close to criticality. We show that the presence of dropout destroys the order-to-chaos critical point and therefore strongly limits the maximum trainable depth for random networks. Finally, we develop a mean field theory for backpropagation and we show that the ordered and chaotic phases correspond to regions of vanishing and exploding gradient respectively.
1 INTRODUCTION
Deep neural network architectures have become ubiquitous in machine learning. The success of deep networks is due to the fact that they are highly expressive (Montufar et al., 2014) while simultaneously being relatively easy to optimize (Choromanska et al., 2015; Goodfellow et al., 2014) with strong generalization properties (Recht et al., 2015). Consequently, developments in machine learning often accompany improvements in our ability to train increasingly deep networks. Despite this, designing novel network architectures is frequently equal parts art and science. This is, in part, because a general theory for neural networks that might inform design decisions has lagged behind the feverish pace of design.
A pair of recent papers (Poole et al., 2016; Raghu et al., 2016) demonstrated that random neural networks are exponentially expressive in their depth. Central to their approach was the consideration of networks after random initialization, whose weights and biases were i.i.d. Gaussian distributed. In particular the paper by Poole et al. (2016) developed a “mean field” formalism for treating wide, untrained, neural networks. They showed that these mean field networks exhibit an order-to-chaos transition as a function of the weight and bias variances. Notably the mean field formalism is not closely tied to a specific choice of activation function or loss.
In this paper, we demonstrate the existence of several characteristic “depth” scales that emerge naturally and control signal propagation in these random networks. We then show that one of these depth scales, ξc, diverges at the boundary between order and chaos. This result is insensitive to many architectural decisions (such as choice of activation function) and will generically be true at any order-to-chaos transition. We then extend these results to include dropout and we show that even small amounts of dropout destroys the order-to-chaos critical point and consequently removes the divergence in ξc. Together these results bound the depth to which signal may propagate through random neural networks.
We then develop a corresponding mean field model for gradients and we show that a duality exists between the forward propagation of signals and the backpropagation of gradients. The ordered and chaotic phases that Poole et al. (2016) identified correspond to regions of vanishing and exploding gradients, respectively. We demonstrate the validity of this mean field theory by computing gradients of random networks on MNIST. This provides a formal explanation of the ‘vanishing gradients’
∗Work done as a member of the Google Brain Residency program (g.co/brainresidency)
phenomenon that has long been observed in neural networks (Bengio et al., 1993). We continue to show that the covariance between two gradients is controlled by the same depth scale that limits correlated signal propagation in the forward direction.
Finally, we hypothesize that a necessary condition for a random neural network to be trainable is that information should be able to pass through it. Thus, the depth-scales identified here bound the set of hyperparameters that will lead to successful training. To test this ansatz we train ensembles of deep, fully connected, feed-forward neural networks of varying depth on MNIST and CIFAR10, with and without dropout. Our results confirm that neural networks are trainable precisely when their depth is not much larger than ξc. This result is dataset independent and is, therefore, a universal function of network architecture.
A corollary of these result is that asymptotically deep neural networks should be trainable provided they are initialized sufficiently close to the order-to-chaos transition. The notion of “edge of chaos” initialization has been explored previously. Such investigations have been both direct as in Bertschinger et al. (2005); Glorot & Bengio (2010) or indirect, through initialization schemes that favor deep signal propagation such as batch normalization (Ioffe & Szegedy, 2015), orthogonal matrix initialization (Saxe et al., 2014), random walk initialization (Sussillo & Abbott, 2014), composition kernels (Daniely et al., 2016), or residual network architectures (He et al., 2015). The novelty of the work presented here is two-fold. First, our framework predicts the depth at which networks may be trained even far from the order-to-chaos transition. While a skeptic might ask when it would be profitable to initialize a network far from criticality, we respond by noting that there are architectures (such as neural networks with dropout) where no critical point exists and so this more general framework is needed. Second, our work provides a formal, as opposed to intuitive, explanation for why very deep networks can only be trained near the edge of chaos.
2 BACKGROUND
We begin by recapitulating the mean-field formalism developed in Poole et al. (2016). Consider a fully-connected, untrained, feed-forward, neural network of depth L with layer width Nl and some nonlinearity φ : R → R. Since this is an untrained neural network we suppose that its weights and biases are respectively i.i.d. as W lij ∼ N(0, σ2w/Nl) and bli ∼ N(0, σ2b ). Notationally we set zli to be the pre-activations of the lth layer and yl+1i to be the activations of that layer. Finally, we take the input to the network to be y0i = xi. The propagation of a signal through the network is described by the pair of equations,
zli = ∑ j W lijy l j + b l i y l+1 i = φ(z l i). (1)
Since the weights and biases are randomly distributed, these equations define a probability distribution on the activations and pre-activations over an ensemble of untrained neural networks. The “mean-field” approximation is then to replace zli by a Gaussian whose first two moments match those of zli. For the remainder of the paper we will take the mean field approximation as given.
Consider first the evolution of a single input, xi;a, as it evolves through the network (as quantified by yli;a and z l i;a). Since the weights and biases are independent with zero mean, the first two moments of the pre-activations in the same layer will be,
E[zli;a] = 0 E[zli;azlj;a] = qlaaδij (2)
where δij is the Kronecker delta. Here qlaa is the variance of the pre-activations in the lth layer due to an input xi;a and it is described by the recursion relation,
qlaa = σ 2 w
∫ Dzφ2 (√ ql−1aa z ) + σ2b (3)
where ∫ Dz = 1√
2π
∫ dze− 1 2 z 2
is the measure for a standard Gaussian distribution. Together these equations completely describe the evolution of a single input through a mean field neural network. For any choice of σ2w and σ 2 b with bounded φ, eq. 3 has a fixed point at q ∗ = liml→∞ qlaa.
The propagation of a pair of signals, x0i;a and x 0 i;b, through this network can be understood similarly. Here the mean pre-activations are trivially the same as in the single-input case. The independence
of the weights and biases implies that the covariance between different pre-activations in the same layer will be given by, E[zli;azlj;b] = qlabδij . The covariance, qlab, will be given by the recurrence relation,
qlab = σ 2 w ∫ Dz1Dz2φ(u1)φ(u2) + σ2b (4)
where u1 = √ ql−1aa z1 and u2 = √ ql−1bb ( cl−1ab z1 + √ 1− (cl−1ab )2z2 ) , with clab = q l ab/ √ qlaaq l bb,
are Gaussian approximations to the pre-activations in the preceding layer with the correct covariance matrix. Moreover clab is the correlation between the two inputs after l layers.
Examining eq. 4 it is clear that c∗ = 1 is a fixed point of the recurrence relation. To determine whether or not the c∗ = 1 is an attractive fixed point the quantity,
χ1 = ∂clab ∂cl−1ab = σ2w
∫ Dz [ φ′ (√ q∗z )]2
(5)
is introduced. Poole et al. (2016) note that the c∗ = 1 fixed point is stable if χ1 < 1 and is unstable otherwise. Thus, χ1 = 1 represents a critical line separating an ordered phase (in which c∗ = 1 and all inputs end up asymptotically correlated) and a chaotic phase (in which c∗ < 1 and all inputs end up asymptotically decorrelated). For the case of φ = tanh, the phase diagram in fig. 1 (a) is observed.
3 ASYMPTOTIC EXPANSIONS AND DEPTH SCALES
Our first contribution is to demonstrate the existence of two depth-scales that arise naturally within the framework of mean field neural networks. Motivating the existence of these depth-scales, we iterate eq. 3 and 4 until convergence for many values of σ2w between 0.1 and 3.0 and with σ 2 b = 0.05 starting with q0aa = q 0 bb = 0.8 and c 0 ab = 0.6. We see, in fig. 1 (b) and (c), that the manner in which both qlaa approaches q ∗ and clab approaches c
∗ is exponential over many orders of magnitude. We therefore anticipate that asymptotically |qlaa − q∗| ∼ e−l/ξq and |clab − c∗| ∼ e−l/ξc for sufficiently large l. Here, ξq and ξc define depth-scales over which information may propagate about the magnitude of a single input and the correlation between two inputs respectively.
We will presently prove that qlaa and c l ab are asymptotically exponential. In both cases we will use the same fundamental strategy wherein we expand one of the recurrence relations (either eq. 3 or eq. 4) about its fixed point to get an approximate “asymptotic” recurrence relation. We find that this asymptotic recurrence relation in turn implies exponential decay towards the fixed point over a depth-scale, ξx.
We first analyze eq. 3 and identify a depth-scale at which information about a single input may propagate. Let qlaa = q ∗ + l. By construction so long as liml→∞ qlaa = q ∗ exists it follows that
l → 0 as l→∞. Eq. 3 may be expanded to lowest order in l to arrive at an asymptotic recurrence relation (see Appendix 7.1),
l+1 = l [ χ1 + σ 2 w ∫ Dzφ′′ (√ q∗z ) φ (√ q∗z )] +O ( ( l)2 ) . (6)
Notably, the term multiplying l is a constant. It follows that for large l the asymptotic recurrence relation has an exponential solution, l ∼ e−l/ξq , with ξq given by
ξ−1q = − log [ χ1 + σ 2 w ∫ Dzφ′′ (√ q∗z ) φ (√ q∗z )] . (7)
This establishes ξq as a depth scale that controls how deep information from a single input may penetrate into a random neural network.
Next, we consider eq. 4. Using a similar argument (detailed in Appendix 7.2) we can expand about clab = c ∗ + l to find an asymptotic recurrence relation,
l+1 = l [ σ2w ∫ Dz1Dz2φ′(u∗1)φ′(u∗2) ] +O(( l)2). (8)
Here u∗1 = √ q∗z1 and u∗2 = √ q∗(c∗z1 + √ 1− (c∗)2z2). Thus, once again, we expect that for large l this recurrence will have an exponential solution, l ∼ e−l/ξc , with ξc given by
ξ−1c = − log [ σ2w ∫ Dz1Dz2φ′(u∗1)φ′(u∗2) ] . (9)
In the ordered phase c∗ = 1 and so ξ−1c = − logχ1. Since the transition between order and chaos occurs when χ1 = 1 it follows that ξc diverges at any order-to-chaos transition so long as q∗ and c∗ exist.
These results can be investigated intuitively by plotting cl+1ab vs c l ab in fig. 2 (a). In the ordered phase there is only a single fixed point, clab = 1. In the chaotic regime we see that a second fixed point develops and the clab = 1 point becomes unstable. We see that the linearization about the fixed points becomes significantly closer to the trivial map near the order-to-chaos transition.
To test these claims we measure ξq and ξc directly by iterating the recurrence relations for qlaa and clab as before with q 0 aa = q 0 bb = 0.8 and c 0 ab = 0.6. In this case we consider values of σ 2 w between
0.1 and 3.0 and σ2b between 0.01 and 0.3. For each hyperparameter settings we fit the resulting residuals, |qlaa − q∗| and |clab − c∗|, to exponential functions and infer the depth-scale. We then compare this measured depth-scale to that predicted by the asymptotic expansion. The result of this measurement is shown in fig. 2. In general we see that the agreement is quite good. As expected we see that ξc diverges at the critical point.
As observed in Poole et al. (2016) we see that the depth scale for the propagation of information in a single input, ξq , is consistently finite and significantly shorter than ξc. To understand why this is the case consider eq. 6 and note that for tanh nonlinearities the second term is always negative. Thus, even as χ1 approaches 1 we expect χ1 + σ2w ∫ Dzφ′′(√q∗z)φ(√q∗z) to be substantially smaller than 1.
3.1 DROPOUT
The mean field formalism can be extended to include dropout. The main contribution here will be to argue that even infinitesimal amounts of dropout destroys the mean field critical point, and therefore limits the trainable network depth. In the presence of dropout the propagation equation, eq. 1, becomes,
zli = 1
ρ ∑ j W lijp l jy l j + b l i (10)
where pj ∼ Bernoulli(ρ) and ρ is the dropout rate. As is typically the case we have re-scaled the sum by ρ−1 so that the mean of the pre-activation is invariant with respect to our choice of dropout rate.
Following a similar procedure to the original mean field calculation consider the fate of two inputs, x0i;a and x 0 i;b, as they are propagated through such a random network. We take the dropout masks to be chosen independently for the two inputs mimicking the manner in which dropout is employed in practice. With dropout the diagonal term in the covariance matrix will be (see Appendix 7.3),
q̄laa = σ2w ρ
∫ Dzφ2 (√ q̄l−1aa z ) + σ2b . (11)
The variance of a single input with dropout will therefore propagate in an identical fashion to the vanilla case with a re-scaling σ2w → σ2w/ρ. Intuitively, this result implies that, for the case of a single input, the presence of dropout simply increases the effective variance of the weights.
Computing the off-diagonal term of the covariance matrix similarly (see Appendix 7.4),
q̄lab = σ 2 w ∫ Dz1Dz2φ(ū1)φ(ū2) + σ2b (12)
with ū1, ū2, and c̄lab defined by analogy to the mean field equations without dropout. Here, unlike in the case of a single input, the recurrence relation is identical to the recurrence relation without dropout. To see that c̄∗ = 1 is no longer a fixed point of these dynamics consider what happens to eq. 12 when we input c̄l = 1. For simplicity, we leverage the short range of ξq to replace q̄laa = q̄ l bb = q̄ ∗. We find (see Appendix 7.5),
c̄l+1ab = 1− 1− ρ ρq̄∗ σ2w
∫ Dzφ2 (√ q̄∗z ) . (13)
The second term is positive for any ρ < 1. This implies that if c̄lab = 1 for any l then c̄ l+1 ab < 1. Thus, c∗ = 1 is not a fixed point of eq. 12 for any ρ < 1. Since eq. 12 is identical in form to eq. 4 it follows that the depth scale for signal propagation with dropout will likewise be given by eq. 9 with the substitutions q∗ → q̄∗ and c∗ → c̄∗ computed using eq. 11 and eq. 12 respectively. Importantly, since there is no longer a sharp critical point with dropout we do not expect a diverging depth scale.
As in networks without dropout we plot, in fig. 3 (a), the iterative map c̄l+1ab as a function of c̄ l ab. Most significantly, we see that the c̄lab = 1 is no longer a fixed point of the dynamics. Instead, as the dropout rate increases c̄lab gets mapped to decreasing values and the fixed point monotonically decreases.
To test these results we plot in fig. 3 (b) the asymptotic correlation, c∗, as a function of σ2w for different values of dropout from ρ = 0.8 to ρ = 1.0. As expected, we see that for all ρ < 1 there is no sharp transition between c∗ = 1 and c∗ < 1. Moreover as the dropout rate increases the correlation c∗ monotonically decreases. Intuitively this makes sense. Identical inputs passed through two different dropout masks will become increasingly dissimilar as the dropout rate increases. In fig. 3 (c) we show the depth scale, ξc, as a function of σ2w for the same range of dropout probabilities. We find that, as predicted, the depth of signal propagation with dropout is drastically reduced and, importantly, there is no longer a divergence in ξc. Increasing the dropout rate continues to decrease the correlation depth for constant σ2w.
4 GRADIENT BACKPROPAGATION
There is a duality between the forward propagation of signals and the backpropagation of gradients. To elucidate this connection consider the backpropagation equations given a loss E,
∂E
∂W lij = δliφ(z l−1 j ) δ l i = φ ′(zli) ∑ j δl+1j W l+1 ji (14)
with the identification δli = ∂E/∂z l i. Within mean field theory, it is clear that the scale of fluctuations of the gradient of weights in a layer will be proportional to E[(δli)2] (see appendix 7.6). In contrast to the pre-activations in forward propagation (eq. 1), the δli will typically not be Gaussian distributed even in the large layer width limit.
Nonetheless, we can work out a recurrence relation for the variance of the error, q̃ laa = E[(δli)2], leveraging the Gaussian ansatz on the pre-activations. In order to do this, however, we must first make an additional approximation that the weights used during forward propagation are drawn independently from the weights used in backpropagation. This approximation is similar in spirit to the vanilla mean field approximation and is reminiscent of work on feedback alignment (Lillicrap et al., 2014). With this in mind we arrive at the recurrence (see appendix 7.7),
q̃ laa = q̃ l+1 aa Nl+1 Nl χ1. (15)
The presence of χ1 in the above equation should perhaps not be surprising. In Poole et al. (2016) they show that χ1 is intimately related to the tangent space of a given layer in mean field neural
networks. We note that the backpropagation recurrence features an explicit dependence on the ratio of widths of adjacent layers of the network, Nl+1/Nl. Here we will consider exclusively constant width networks where this factor is unity. For a discussion of the case of unequal layer widths see Glorot & Bengio (2010).
Since χ1 depends only on the asymptotic q∗ it follows that for constant width networks we expect eq. 15 to again have an exponential solution with,
q̃ laa = q̃ L aae −(L−l)/ξ∇ ξ−1 ∇ = − logχ1. (16)
Note that here ξ−1 ∇ = − logχ1 both above and below the transition. It follows that ξ∇ can be both positive and negative. We conclude that there should be three distinct regimes for the gradients.
1. In the ordered phase, χ1 < 1 and so ξ∇ > 0. We therefore expect gradients to vanish over a depth |ξ∇ |. 2. At criticality, χ1 → 1 and so ξ∇ →∞. Here gradients should be stable regardless of depth. 3. In the chaotic phase, χ1 > 1 and so ξ∇ < 0. It follows that in this regime gradients should
explode over a depth |ξ∇ |.
Intuitively these three regimes make sense. To see this, recall that perturbations to a weight in layer l can alternatively be viewed as perturbations to the pre-activations in the same layer. In the ordered phase both the perturbed signal and the unperturbed signal will be asymptotically mapped to the same point and the derivative will be small. In the chaotic phase the perturbed and unperturbed signals will become asymptotically decorrelated and the gradient will be large.
To investigate these predictions we construct deep random networks of depth L = 240 and layerwidth Nl = 300. We then consider the cross-entropy loss of these networks on MNIST. In fig. 4 (a) we plot the layer-by-layer 2-norm of the gradient, ||∇W labE|| 2 2, as a function of layer, l, for different values of σ2w. We see that ||∇W labE|| 2 2 behaves exponentially over many orders of magnitude. Moreover, we see that the gradient vanishes in the ordered phase and explodes in the chaotic phase. We test the quantitative predictions of eq. 16 in fig. 4 (b) where we compare |ξ∇ | as predicted from theory with the measured depth-scale constructed from exponential fits to the gradient data. Here we see good quantitative agreement between the theoretical predictions from mean field random networks and experimentally realized networks. Together these results suggest that the approximations on the backpropagation equations were representative of deep, wide, random networks.
Finally, we show that the depth scale for correlated signal propagation likewise controls the depth at which information stored in the covariance between gradients can survive. The existence of
consistent gradients across similar samples from a training set ought to be especially important for determining whether or not a given neural network architecture can be trained. To establish this depth-scale first note (see Appendix 7.8) that the covariance between gradients of two different inputs, xi;1 and xi;2, will be proportional to (∇W lijEa) · (∇W lijEb) ∼ E[δ l i;aδ l i;b] = q̃ l ab where Ea is the loss evaluated on xi;a and δi;a = ∂Ea/∂zli;a are appropriately defined errors.
It can be shown (see Appendix 7.9) that q̃ lab features the recurrence relation,
q̃ lab = q̃ l+1 ab Nl+1 Nl+2 σ2w
∫ Dz1Dz2φ′(u1)φ′(u2) (17)
where u1 and u2 are defined similarly as for the forward pass. Expanding asymptotically it is clear that to zeroth order in l, q̃lab will have an exponential solution with q̃ l ab = q̃ L abe −(L−l)/ξc with ξc as defined in the forward pass.
5 EXPERIMENTAL RESULTS
Taken together, the results of this paper lead us to the following hypothesis: a necessary condition for a random network to be trained is that information about the inputs should be able to propagate forward through the network, and information about the gradients should be able to propagate backwards through the network. The preceding analysis shows that networks will have this property precisely when the network depth, L, is not much larger than the depth-scale ξc. This criterion is data independent and therefore offers a “universal” constraint on the hyperparameters that depends on network architecture alone. We now explore this relationship between depth of signal propagation and network trainability empirically.
To investigate this prediction, we consider random networks of depth 10 ≤ L ≤ 300 and 1 ≤ σ2w ≤ 4 with σ2b = 0.05. We train these networks using Stochastic Gradient Descent (SGD) and RMSProp
on MNIST and CIFAR10. We use a learning rate of 10−3 for SGD when L . 200, 10−4 for larger L, and 10−5 for RMSProp. These learning rates were selected by grid search between 10−6 and 10−2 in exponentially spaced steps of size 10. We note that the depth dependence of learning rate was explored in detail in Saxe et al. (2014). In fig. 5 (a)-(d) we color in red the training accuracy that neural networks achieved as a function of σ2w and L for different datasets, training time, and choice of minimizer (see Appendix 7.10 for more comparisons). In all cases the neural networks over-fit the data to give a training accuracy of 100% and test accuracies of 98% on MNIST and 55% on CIFAR10. We emphasize that the purpose of this study is to demonstrate trainability as opposed to optimizing test accuracy.
We now make the connection between the depth scale, ξc, and the maximum trainable depth more precise. Given the arguments in the preceding sections we note that if L = nξc then signal through the network will be attenuated by a factor of en. To understand how much signal can be lost while still allowing for training, we overlay in fig. 5 (a) curves corresponding to nξc from n = 1 to 6. We find that networks appear to be trainable when L . 6ξc. It would be interesting to understand why this is the case.
Motivated by this argument in fig. 5 (b)-(d) in white, dashed, overlay we plot twice the predicted depth scale, 6ξc. There is clearly a relationship between the depth of correlated signal propagation and whether or not these networks are trainable. Networks closer to their critical point appear to train more quickly than those further away. Moreover, this relationship has no obvious dependence on dataset, duration of training, or minimizer. We therefore conclude that these bounds on trainable hyperparameters are universal. This in turn implies that to train increasingly deep networks, one must generically be ever closer to criticality.
Next we consider the effect of dropout. As we showed earlier, even infinitesimal amounts of dropout disrupt the order-to-chaos phase transition and cause the depth scale to become finite. However, since the effect of a single dropout mask is to simply re-scale the weight variance by σ2w → σ2w/ρ, the gradient magnitude will be stable near criticality, while the input and gradient correlations will not be. This therefore offers a unique opportunity to test whether the relevant depth-scale is |1/ logχ1| or ξc.
In fig. 6 we repeat the same experimental setup as above on MNIST with dropout rates ρ = 0.99, 0.98, and 0.94. We observe, first and foremost, that even extremely modest amounts of dropout limit the maximum trainable depth to about L = 100. We additionally notice that the depth-scale, ξc, predicts the trainable region accurately for varying amounts of dropout.
6 DISCUSSION
In this paper we have elucidated the existence of several depth-scales that control signal propagation in random neural networks. Furthermore, we have shown that the degree to which a neural network can be trained depends crucially on its ability to propagate information about inputs and gradients
through its full depth. At the transition between order and chaos, information stored in the correlation between inputs can propagate infinitely far through these random networks. This in turn implies that extremely deep neural networks may be trained sufficiently close to criticality. However, our contribution goes beyond advocating for hyperparameter selection that brings random networks to be nearly critical. Instead, we offer a general purpose framework that predicts, at the level of mean field theory, which hyperparameters should allow a network to be trained. This is especially relevant when analyzing schemes like dropout where there is no critical point and which therefore imply an upper bound on trainable network depth.
An alternative perspective as to why information stored in the covariance between inputs is crucial for training can be understood by appealing to the correspondence between infinitely wide Bayesian neural networks and Gaussian Processes (Neal, 2012). In particular the covariance, qlab, is intimately related to the kernel of the induced Gaussian Process. It follows that cases in which signal stored in the covariance between inputs may propagate through the network correspond precisely to situations in which the associated Gaussian Process is well defined.
Our work suggests that it may be fruitful to investigate pre-training schemes that attempt to perturb the weights of a neural network to favor information flow through the network. In principle this could be accomplished through a layer-by-layer local criterion for information flow or by selecting the mean and variance in schemes like batch normalization to maximize the covariance depth-scale.
These results suggest that theoretical work on random neural networks can be used to inform practical architectural decisions. However, there is still much work to be done. For instance, the framework developed here does not apply to unbounded activations, such as rectified linear units, where it can be shown that there are phases in which eq. 3 does not have a fixed point. Additionally, the analysis here applies directly only to fully connected feed-forward networks, and will need to be extended to architectures with structured weight matrices such as convolutional networks.
We close by noting that in physics it has long been known that, through renormalization, the behavior of systems near critical points can control their behavior even far from the idealized critical case. We therefore make the somewhat bold hypothesis that a broad class of neural network topologies will be controlled by the fully-connected mean field critical point.
ACKNOWLEDGMENTS
We thank Ben Poole, Jeffrey Pennington, Maithra Raghu, and George Dahl for useful discussions. We are additionally grateful to RocketAI for introducing us to Temporally Recurrent Online Learning and two-dimensional time.
7 APPENDIX
Here we present derivations of results from throughout the paper.
7.1 SINGLE INPUT DEPTH-SCALE
Result:
Consider the recurrence relation for the variance of a single input,
qlaa = σ 2 w
∫ Dzφ2 (√ ql−1aa z ) + σ2b (18)
and a fixed point of the dynamics, q∗. qlaa can be expanded about the fixed point to yield the asymptotic recurrence relation,
l+1 = l [ χ1 + σ 2 w ∫ Dzφ′′ (√ q∗z ) φ (√ q∗z )] +O ( ( l)2 ) . (19)
Derivation:
We begin by first expanding to order l,
q∗ + l+1 = σ2w
∫ Dz [ φ (√ q∗ + lz )]2
+ σ2b (20)
≈ σ2w ∫ Dz [ φ (√ q∗z + 1
2 lz√ q∗
)]2 + σ2b (21)
≈ σ2w ∫ Dz [ φ (√ q∗z ) + 1
2 lz√ q∗ φ′ (√ q∗z )]2 + σ2b +O(( l)2) (22)
≈ σ2w ∫ Dzφ2 (√ q∗z ) + σ2b + l σ
2 w√ q∗
∫ Dzzφ (√ q∗z ) φ′ (√ q∗z ) +O(( l)2) (23)
≈ q∗ + l σ 2 w√ q∗
∫ Dzzφ(√q∗z)φ′ (√ q∗z ) +O(( l)2). (24)
We therefore arrive at the approximate reccurence relation,
l+1 = l σ2w√ q∗
∫ Dzzφ(√q∗z)φ′ (√ q∗z ) +O(( l)2). (25)
Using the identity, ∫ Dzzf(z) = ∫ Dzf ′(z) we can rewrite this asymptotic recurrence relation as,
l+1 = l [ σ2w ∫ Dz [ φ′ (√ q∗z )]2 + σ2w ∫ Dzφ′′ (√ q∗z ) φ (√ q∗z )] +O(( l)2) (26)
= l [ χ1 + σ 2 w ∫ Dzφ′′ (√ q∗z ) φ (√ q∗z )] +O(( l)2) (27)
as required.
7.2 TWO INPUT DEPTH-SCALE
Result:
Consider the recurrence relation for the co-variance of two input,
qlab = σ 2 w ∫ Dz1Dz2φ(u1)φ(u2) + σ2b , (28)
a correlation between the inputs, clab = q l ab/ √ qlaaq l bb, and a fixed point of the dynamics, c
∗. clab can be expanded about the fixed point to yield the asymptotic recurrence relation,
l+1 = l [ σ2w ∫ Dz1Dz2φ′(u1)φ′(u2) ] +O ( ( l)2 ) . (29)
Derivation:
Since the relaxation of qlaa and q l bb to q ∗ occurs much more quickly than the convergence of qlab we approximate qlaa = q l bb = q
∗ as in Poole et al. (2016). We therefore consider the perturbation qlab/q ∗ = clab = c ∗ + l. It follows that we may make the approximation,
ul2 = √ q∗ ( clabz1 + √ 1− (clab)2z2 ) (30)
≈ √q∗ ( c∗z1 + √ 1− (c∗)2 − 2c∗ lz2 ) + √ q∗ lz1 +O( 2) (31)
(32)
We now consider the case where c∗ < 1 and c∗ = 1 separately; we will later show that these two results agree with one another. First we consider the case where c∗ < 1 in which case we may safely expand the above equation to get,
ul2 = √ q∗ ( c∗z1 + √ 1− (c∗)2z2 ) + √ q∗ l ( z1 −
c∗√ 1− (c∗)2 z2
) +O( 2). (33)
This allows us to in turn approximate the recurrence relation,
cl+1ab = σ2w q∗
∫ Dz1Dz2φ(u∗1)φ(ul2) + σ2b (34)
≈ σ 2 w
q∗
∫ Dz1Dz2φ(u∗1) [ φ(u∗2) + √ q∗ l ( z1 −
c∗√ 1− (c∗)2 z2
) φ′(u∗2) ] + σ2b +O( 2)
(35) = c∗ + σ2w√ q∗ l ∫ Dz1Dz2 ( z1 − c∗√ 1− (c∗)2 z2 ) φ(u∗1)φ ′(u∗2) (36)
= c∗ + σ2w√ q∗ l
[∫ Dz1Dz2z1φ(u∗1)φ′(u∗2)−
c∗√ 1− (c∗)2
∫ Dz1Dz2z2φ(u∗1)φ′(u∗2) ] (37)
= c∗ + σ2w l [∫ Dz1Dz2(φ′(u∗1)φ′(u∗2) + c∗φ(u∗1)φ′′(u∗2))− c∗ ∫ Dz1Dz2φ(u∗1)φ′′(u∗2) ] (38)
= c∗ + σ2w l ∫ Dz1Dz2φ′(u∗1)φ′(u∗2). (39)
where u∗1 and u ∗ 2 are appropriately defined asymptotic random variables. This leads to the asymptotic recurrence relation,
l+1 = σ2w l ∫ Dz1Dz2φ′(u∗1)φ′(u∗2) (40)
as required.
We now consider the case where c∗ = 1 and clab = 1 − l. In this case the expansion of ul2 will become,
ul2 = √ q∗z1 + √ 2q∗ lz2 − √ q∗ lz1 +O( 3/2) (41)
and so the lowest order correction is of orderO( √ l) as opposed toO( l). As usual we now expand the recurrence relation, noting that u∗2 = u ∗ 1 is independent of z2 when c ∗ = 1 to find,
cl+1ab = σ2w q∗
∫ Dz1Dz2φ(u∗1)φ(ul2) + σ2b (42)
≈ σ 2 w
q∗
∫ Dz1Dz2φ(u∗1) [ φ(u∗2) + (√ 2q∗ lz2 − √ q∗ lz1 ) φ′(u∗2) + q ∗ lz22φ ′′(u∗2) ] + σ2b
(43)
= c∗ + σ2w l
∫ Dzφ(√q∗z) [ φ′′( √ q∗z)− 1√
q∗ zφ′( √ q∗z)
] (44)
= c∗ + σ2w l
[∫ Dzφ(√q∗z)φ′′(√q∗z)− 1√
q∗
∫ Dzzφ(√q∗z)φ′(√q∗z) ] (45)
= c∗ − σ2w l ∫ Dz [ φ′( √ q∗z) ]2 (46)
It follows that the asymptotic recurrence relation in this case will be, l+1 = − lσ2w ∫ Dz [ φ′( √ q∗z) ]2 = − lχ1. (47)
where χ1 is the stability condition for the ordered phase. We note that although the approximations were somewhat different the asymptotic recurrence relation for c∗ < 1 reduces eq. 47 result for c∗ = 1. We may therefore use 4 for all c∗.
7.3 VARIANCE OF AN INPUT WITH DROPOUT
Result:
In the presence of dropout with rate ρ, the variance of a single input as it is passed through the network is described by the recurrence relation,
q̄laa = σ2w ρ
∫ Dzφ2 (√ q̄l−1aa z ) + σ2b . (48)
Derivation:
Recall that the recurrence relation for the pre-activations is given by,
zli = 1
ρ ∑ j W lijp l jy l j + b l i (49)
where plj ∼ Bernoulli(ρ). It follows that the variance will be given by,
q̄laa = E[(zli)2] (50)
= 1
ρ2 ∑ j E[(W lij)2]E[(ρlj)2]E[(ylj)2] + E[(bli)2] (51)
= σ2w ρ
∫ Dzφ2 (√ q̄l−1aa z ) + σ2b . (52)
where we have used the fact that E[(plj)2] = ρ.
7.4 COVARIANCE OF TWO INPUTS WITH DROPOUT
Result:
The co-variance between two signals, zli;a and z l i;b, with separate i.i.d. dropout masks p l i;a and p l i;b is given by,
q̄lab = σ 2 w ∫ Dz1Dz2φ(ū1)φ(ū2) + σ2b . (53)
where, in analogy to eq. 4, ū1 = √ q̄laaz1 and ū2 = √ q̄lbb ( c̄labz1 + √ 1− (c̄lab)2z2 ) .
Derivation:
Proceeding directly we find that,
E[zli;azli;b] = 1
ρ2 ∑ j E[(W lij)2]E[plj;a]E[plj;b]E[ylj;aylj;b] + E[bli] (54)
= σ2w ∫ Dz1Dz2φ(ū1)φ(ū2) + σ2b (55)
where we have used the fact that E[pli;a] = E[pli;b] = ρ. We have also used the same substitution for E[ylj;aylj;b] used in the original mean field calculation with the appropriate substitution.
7.5 THE LACK OF A c∗ = 1 FIXED POINT WITH DROPOUT
Result:
If clab = 1 then it follows that,
c̄l+1ab = 1− 1− ρ ρq̄∗ σ2w
∫ Dzφ2 (√ q̄∗z )
(56)
subject to the approximation, qlaa ≈ qlbb ≈ q∗. This implies that cl+1ab < 1. Derivation:
Plugging in clab = 1 with q l aa ≈ qlbb ≈ q∗ we find that ū1 = ū2 = √ q∗z1. It follows that,
cl+1ab = ql+1ab q∗
(57)
= 1
q∗
[ σ2w ∫ Dzφ2 (√ q∗z ) + σ2b ] (58)
= 1
q∗
[ σ2w(1− ρ−1 + ρ−1) ∫ Dzφ2 (√ q∗z ) + σ2b ] (59)
= 1
q∗ [ σ2w ρ ∫ Dzφ2 (√ q∗z ) + σ2b ] + σ2w q∗ (1− ρ−1) ∫ Dzφ2 (√ q∗z )
(60)
= 1− 1− ρ ρq̄∗ σ2w
∫ Dzφ2 (√ q̄∗z )
(61)
as required. Here we have integrated out z2 since nether ū1 nor ū2 depend on it.
7.6 MEAN FIELD GRADIENT SCALING
Result:
In mean field theory the expected magnitude of the gradient ||∇W lijE|| 2 will be proportional to E[(δli)2].
Derivation:
We first note that since the W lij are i.i.d. it follows that,
||∇W lijE|| 2 = ∑ ij
( ∂E
∂W lij
)2 (62)
≈ NlNl+1E ( ∂E ∂W lij )2 (63) where we have used the fact that the first line is related to the sample expectation over the different realizations of theW lij to approximate it by the analytic expectation in the second line. In mean field theory since the pre-activations in each layer are assumed to be i.i.d. Gaussian it follows that,
E ( ∂E ∂W lij )2 = E[(δli)2]E[φ2(zl−1j )] (64) and the result follows.
7.7 MEAN FIELD BACKPROPAGATION
Result:
In mean field theory the recursion relation for the variance of the errors, q̃ l = E[(δli)2] is given by,
q̃ laa = q̃ l+1 aa Nl+1 Nl+2 χ1(q l aa). (65)
Derivation:
Computing the variance directly and using mean field approximation, q̃ laa = E[(δli;a)2] = E[(φ′(zli;a))2] ∑ j E[(δl+1j;a ) 2]E[(W l+1ji ) 2] (66)
= E[(φ′(zli;a))2] σ2w Nl+1 ∑ j E[(δl+1j;a ) 2] (67)
= E[(φ′(zli;a))2] Nl+1 Nl+2 σ2w q̃ l+1 aa (68)
= σ2w q̃ l+1 aa Nl+1 Nl+2
∫ Dz [ φ′ (√
qlaaz
)]2 (69)
≈ q̃ l+1aa Nl+1 Nl+2 χ1 (70)
as required. In the last step we have made the approximation that qlaa ≈ q∗ since the depth scale for the variance is short ranged.
7.8 MEAN FIELD GRADIENT COVARIANCE SCALING
Result:
In mean field theory we expect the covariance between the gradients of two different inputs to scale as,
(∇W lijEa) · (∇W lijEb) ∼ E[δi;aδi;b]. (71)
Derivation:
We proceed in a manner analogous to Appendix 7.6. Note that in mean field theory since the weights are i.i.d. it follows that
(∇W lijEa) · (∇W lijEb) = ∑ ij ∂Ea ∂W lij ∂Eb ∂W lij
(72)
≈ NlNl+1E [ ∂Ea ∂W lij ∂Eb ∂W lij ] (73)
where, as before, the final term is approximating the sample expectation. Since the weights in the forward and backwards passes are chosen independently it follows that we can factor the expectation as,
E [ ∂Ea ∂W lij ∂Eb ∂W lij ] = E[δli;aδli;b]E[φ(zli;a)φ(zli;b)] (74)
and the result follows.
7.9 MEAN FIELD BACKPROPAGATION OF COVARIANCE
Result:
The covariance between the gradients due to two inputs scales as,
q̃ lab = q̃ l+1 ab Nl+1 Nl+2 σ2w
∫ Dz1Dz2φ′(u1)φ′(u2) (75)
under backpropagation.
Derivation
As in the analogous derivation for the variance, we compute directly, q̃ lab = E[δli;aδli;b] = E [φ′(zi;a)φ′(zi;b)] ∑ j E[δl+1j;a δ l+1 j;b ]E[(W l+1 ji ) 2] (76)
= q̃ l+1ab Nl+1 Nl+2 σ2w
∫ Dz1Dz2φ′(u1)φ′(u2) (77)
as required.
7.10 FURTHER EXPERIMENTAL RESULTS
Here we include some more experimental figures that investigate the effects of training time, minimizer, and dataset more closely.

I'm not familiar enough with mean-field techniques to judge the soundness of Eq 2, but I'm willing to roll with it.
Minor point on presentation: Speaking of the "evolution" of x_{i;a} as it travels through the network could give some readers helpful intuition, but for me it was confusing because x_{*;a} is the immutable input vector, and it's the just-introduced z and y variables that represent its so-called evolution, no?
In interpreting this analysis - A network may be trainable if information does not pass through it, if the training steps, by whatever reason, perturb the weights so that information starts to pass through it (without subsequently perturbing the weights to stop information from passing through it.) Perhaps this could be clarified by a definition of “training algorithm”?
Comments on central claims:
Previous work on initializing neural networks to promote information flow (e.g. Glorot & Bengio,


We introduce Information Dropout, a generalization of dropout that is motivated by the Information Bottleneck principle and highlights the way in which injecting noise in the activations can help in learning optimal representations of the data. Information Dropout is rooted in information theoretic principles, it includes as special cases several existing dropout methods, like Gaussian Dropout and Variational Dropout, and, unlike classical dropout, it can learn and build representations that are invariant to nuisances of the data, like occlusions and clutter. When the task is the reconstruction of the input, we show that the information dropout method yields a variational autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Our experiments validate the theoretical intuitions behind our method, and we find that information dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.
1 INTRODUCTION
In the general supervised setting, we want to learn the conditional distribution p(y|x) of some random variable y, which we refer to as the task, given (samples of the) input data x. In typical applications, x is often high dimensional (for example an image or a video), while y is low dimensional, such as a label or a coarsely-quantized location. In such cases, a large part of the variability in x is actually due to nuisance factors that affect the data, but are otherwise irrelevant for the task. Since by definition these nuisance factors are not predictive of the task, they should be disregarded during the inference process. However, it often happens that modern machine learning algorithms, in part due to their high flexibility, will fit spurious correlations, present in the training data, between the nuisances and the task, thus leading to poor generalization performance.
In view of this, Tishby & Zaslavsky (2015) argue that the success of deep learning is in part due to the capability of neural networks to build incrementally better representations that expose the relevant information, while at the same time discarding nuisance variability. This interpretation is intriguing, as it establishes a connection between machine learning, probabilistic inference, and information theory. However, the commonly used training methods do not seem to stem from this insight, and indeed deep networks may maintain even in the topmost layers dependencies on easily ignorable nuisances (see for example Figure 2).
To bring the practice in line with the theory, and to better understand these connections, we introduce a new layer, that we call Information Dropout, which allows the network to selectively introduce multiplicative noise in the layer activations, and thus to control the flow of information. We then introduce a modified cost function, that can be seen as an approximation of the Information Bottleneck Lagrangian of Tishby et al. (1999), which encourages the creation of representations of the data which are increasingly insensitive to the action of nuisances. In practice, this prevents the network from overfitting to the nuisances. As we show in various experiments, our method improves the
∗Dedicated to Naftali Tishby in the occasion of the conference Information, Control and Learning held in his honor in Jerusalem, September 26-28, 2016.
generalization performance, and is comparable or better than the dropout baseline. In particular, it considerably improves over binary dropout on smaller models, since, unlike dropout, Information Dropout also adapts the noise to the structure of the network and to the individual sample at test time.
Apart from the practical interest of Information Dropout, one of our main results is that Information Dropout can be seen as a generalization to several existing dropout methods, providing a unified framework to analyze them, together with some additional insights on empirical results. As we discuss in Section 2, the introduction of noise to prevent overfitting has already been studied from several points of view. For example the original formulation of dropout of Srivastava et al. (2014), which introduces binary multiplicative noise, was motivated as a way of efficiently training an ensemble of exponentially many networks, that would be averaged at testing time. Kingma et al. (2015) introduce Variational Dropout, a dropout method which closely resemble ours, and is instead derived from a Bayesian analysis of neural networks. Information Dropout gives an alternative information-theoretic interpretation of those methods.
As we show in Section 5, other than being very closely related to Variational Dropout, Information Dropout directly yields a variational autoencoder as a special case when the task is the reconstruction of the input. This result is in part expected, since by construction Information Dropout seeks an optimal representation of the input for the task of reconstruction, and the representation given by the latent variables of a variational autoencoder fits the criteria. However, it still rises the question of exactly what and how deep are the links between information theory, representation learning, variational inference and nuisance invariance. This work can be seen as a small step in answering this question.
2 RELATED WORK
The main contribution of our work is to establish how two seemingly different areas of research, namely the study of noise and dropout methods to prevent overfitting, and the study of optimal representations, can be linked through the Information Bottleneck principle.
Dropout was introduced by Srivastava et al. (2014). The original motivation was that by randomly dropping the activations during training, we can effectively train an ensemble of exponentially many networks, that are then averaged during testing, therefore reducing overfitting. Wang & Manning (2013) suggested that dropout could be seen as performing a Monte-Carlo approximation of an implicit loss function, and suggest that instead of multiplying the activations by binary noise, like in the original dropout, multiplicative Gaussian noise with mean 1 can be used as a way of better approximating the implicit loss function. This leads to a comparable performance but faster training than binary dropout.
Kingma et al. (2015) take a similar view of dropout as introducing multiplicative (Gaussian) noise, but instead study the problem from a Bayesian point of view. Given a training dataset D = {(xi,yi)}i=1,...,N and a prior distribution p(w), we want to compute the posterior distribution p(w|D) of the weights w of the network. As is customary in variational inference, the true posterior can be approximated by minimizing the negative variational lower bound L(θ) of the marginal log-likelihood of the data,
L(θ) = 1 N N∑ i=1 Ew∼pθ(w|D)[− log p(yi|xi,w)] + 1 N KL(pθ(w|D) ‖ p(w)).
This minimization is difficult to perform, since it requires to repeatedly sample new weights for each sample of the dataset. As an alternative, Kingma et al. (2015) suggest that the uncertainty about the weights that is expressed by the posterior distribution pθ(w|D) can equivalently be encoded as a multiplicative noise in the activations of the layers (the so called local reparametrization trick). As we will see in the following sections, this loss function closely resemble the one of Information Dropout, which however is derived from a purely information theoretic argument based on the Information Bottleneck principle. One difference is that we allow the parameters of the noise to change on a per-sample basis (which, as we show in the experiments, can be useful to deal with nuisances), and that we allow a scaling constant β in front of the KL-divergence term, which can be changed freely. Interestingly, even if the Bayesian derivation does not allow a rescaling of the KL-divergence, Kingma et al. (2015) notice that choosing a different scale for the KL-divergence term can indeed lead to improvements in practice.
The interpretation of deep neural network as a way of creating successively better representations of the data has already been suggested and explored by many, as described in Tishby & Zaslavsky (2015). Some have focused on creating representations that are maximally invariant to nuisances, especially when they have the structure of a (possibly infinite-dimensional) group acting on the data, like Sundaramoorthi et al. (2009), or, when the nuisance is a locally compact group acting on each layer, by successive approximations implemented by hierarchical convolutional architectures, like Anselmi et al. (2016) and Bruna & Mallat (2011). In these cases, which cover common nuisances such as translations and rotations of an image (affine group), or small diffeomorphic deformations due to a slight change of point of view (group of diffeomorphisms), the representation is equivalent to the data modulo the action of the group. However, when the nuisances are not a group, as is the case for occlusions, it is not possible to achieve such equivalence, that is, there is a loss. To address this problem, Soatto & Chiuso (2016) defined optimal representations not in terms of maximality, but in terms of sufficiency, and characterized representations that are both sufficient and invariant. They argue that the management of nuisance factors common in visual data, such as change of viewpoint, local deformations, and changes of illumination, is directly tied to the specific structure of deep convolutional networks, where local marginalization of simple nuisances at each layer results in marginalization of complex nuisances in the network as a whole.
Our work fits in this last line of thinking, where the goal is not equivalence to the data up to the action of (group) nuisances, but instead sufficiency for the task. Our main contribution in this sense is to show that injecting noise into the layers, and therefore using a non-deterministic function of the data, can actually simplify the theoretical analysis and lead to improved invariance to nuisances. This is an alternate explanation to that put forth by the references above.
3 OPTIMAL REPRESENTATIONS AND THE INFORMATION BOTTLENECK LOSS
Given some input data x, we want to compute some (possibly nondeterministic) function of x, called a representation, that has some desirable properties in view of the task y, for instance by being more convenient to work with, exposing relevant statistics, or being easier to store. Ideally, we want this representation to be as good as the original data for the task, and not squander resources modeling parts of the data that are irrelevant to the task. Formally, this means that we want to find a random variable z satisfying the following conditions:
(i) z is a representation of x; that is, its distribution depends only on x, as expressed by the following Markov chain:
y x z
(ii) z is sufficient for the task y, that is I(x;y) = I(z;y), expressed by the Markov chain:
y z x
(iii) among all random variables satisfying these requirements, the mutual information I(x; z) is minimal. This means that z discards all the variability that is not relevant to the task.
Using the identity I(x;y)− I(z;y) = H(y|z)−H(y|x), where H denotes the entropy and I the mutual information, it is easy to see that the above conditions are equivalent to finding a distribution p(z|x) which solves the optimization problem
minimize I(x; z)
s.t. H(y|z) = H(y|x).
Since the above minimization is difficult to perform in general due to the non-linear constraint, Tishby et al. (1999) propose to minimize instead the following relaxation, known as the Information Bottleneck Lagrangian, L = H(y|z) + βI(x; z). (1) where β is some positive constant. It is easy to see that, in the limit β → 0+, this is equivalent to the original problem. When all random variables are discrete and z = T (x) is a deterministic function of x, the algorithm proposed by Tishby et al. (1999) can be used to minimize the IB Lagrangian
efficiently. However, no algorithm is known to minimize the IB Lagrangian for non-Gaussian, high-dimensional continuous random variables.
Our main result is that, when we restrict to the family of distributions obtained by injecting noise to one layer of a neural network, we can efficiently approximate and minimize the IB Lagrangian. Since we restrict the family of distributions, this does not in general guarantee that the resulting representation will be optimal. We can however iterate the process to obtain incrementally improved representations. As we will show, this process can be effectively implemented through a generalization of the dropout layer that we call Information Dropout.
In preparation for this, we rewrite the IB Lagrangian as a per-sample loss function. Let p(x,y) denote the true distribution of the data, from which the training set {(xi,yi)}i=1,...,N is sampled, and let pθ(z|x) and pθ(y|z) denote the unknown distributions that we need to estimate, parametrized by θ. Then, we can write the two terms in the IB Lagrangian as
H(y|z) ' Ex,y∼p(x,y) [ Ez∼pθ(z|x)[− log pθ(y|z)] ] I(x; z) = Ex∼p(x)[KL(pθ(z|x) ‖ pθ(z))],
whereKL denotes the Kullback-Leibler divergence. We can therefore approximate the IB Lagrangian empirically as
L = 1 N N∑ i=1 Ez∼p(z|xi)[− log p(yi|z)] + βKL(pθ(z|xi) ‖ pθ(z)). (2)
Notice that the first term simply is the average cross-entropy loss, which is the most commonly used loss function in deep learning. The second term can then be seen as a regularization term that penalizes the transfer of information from x to z. In the next section, we discuss ways to control such information transfer through the injection of noise.
Remark. Aside from being easier to work with, stochastic representations of the data can realize a lower value of the IB Lagrangian than any deterministic representation. For an example, consider the task of reconstructing single random bit y given a noisy observation x. The only deterministic representations are equivalent to the either the noisy observation itself or to the trivial constant map. It is not difficult to check that for opportune values of β and of the noise, neither realize the optimal tradeoff reached by an opportune stochastic representation.
4 INFORMATION DROPOUT
Inspired by the effectiveness of dropout, we propose the following way of constructing a representation z: first, we compute a function f(x) of the input through a sequence of convolutional or fullyconnected layers, followed by a nonlinear activation function. Ideally, this should help “disentangle” the nuisances from the (random) function of the data that is relevant to the task, as we illustrate in Appendix C. Subsequently, we selectively let the relevant information flow by applying multiplicative noise from a noise distribution pα(x)(ε) with mean 1 and whose parameters α(x) = αθ(x), such as the variance, are selected by the network itself:
ε ∼ pα(x)(ε), z = ε f(x),
where “ ” denotes the element-wise product. Notice that, if pα(x)(ε) is a Bernoulli distribution rescaled to have mean 1, this reduces exactly to the classic binary dropout layer. As we discussed in Section 2, there are also variants of dropout that use different distributions.
x f(x) z = ε f(x)
ε
y
A natural choice for the multiplicative noise distribution pα(x)(ε), which also simplifies the theoretical analysis, is the log-normal distribution pα(x)(ε) = logN (0, α2θ(x)). Once we fix this noise distribution, given the above expression for z, we can easily compute the distribution pθ(z|x) that appears in eq. (2). However, to be able to compute the KL-divergence term, we still need to fix a prior distribution pθ(z). The choice of this prior largely depends on the expected distribution of the activations f(x). In the following, we assume for simplicity that all the activations are approximately independent and identically distributed. In Appendix B we show that, even if the activations are not independent, optimizing the loss in Equation (2) under the assumption of a factorized prior still makes sense, and it actually encourages the components to become independent. We concentrate on two of the most common activation functions, the rectified linear unit (ReLU), which is easy to compute and works well in practice, and the Softplus function, which can be seen as a strictly positive and differentiable approximation of ReLU.
A network implemented using only ReLU and a final Softmax layer has the remarkable property of being scale-invariant, meaning that multiplying all weights, biases, and activations by a constant does not change the final result. Therefore, from a theoretical point of view, it would be desirable to use a scale-invariant prior. The only such prior is the improper log-uniform, p(log(z)) = c, or equivalently p(z) = c/z, which was also suggested by Kingma et al. (2015), but as a prior for the weights of the network, rather than the activations. Since the ReLU activations are frequently zero, we also assume p(z = 0) = q for some constant 0 ≤ q ≤ 1. Therefore, the final prior has the form p(z) = qδ0(z) + c/z, where δ0 is the Dirac delta in zero. In Figure 1a, we compare this prior distribution with the actual empirical distribution p(z) of a network with ReLU activations.
In a network implemented using Softplus activations, a log-normal is a good fit of the distribution of the activations. This is to be expected, especially when using batch-normalization, since the pre-activations will approximately follow a normal distribution with zero mean, and the Softplus approximately resembles a scaled exponential near zero. Therefore, in this case we suggest using a log-normal distribution as our prior p(z). In Figure 1b, we compare this prior with the empirical distribution p(z) of a network with Softplus activations.
Using these priors, we can finally compute the KL divergence term in eq. (2) for both ReLU activations and Softplus activations. We prove the following two propositions in Appendix A. Proposition 1 (Information dropout cost for ReLU activations). Let z = ε · f(x), where ε ∼ pα(ε), and assume p(z) = qδ0(z) + c/z. Then, assuming f(x) 6= 0, we have
KL(pθ(z|x) ‖ p(z)) = −H(pα(x)(log ε)) + log c
In particular, if pα(ε) is chosen to be the log-normal distribution pα(ε) = logN (0, α2θ(x)), we have
KL(pθ(z|x) ‖ p(z)) = − logαθ(x) + const. (3)
If instead f(x) = 0, we have KL(pθ(z|x) ‖ p(z)) = − log q.
Proposition 2 (Information dropout cost for Softplus activations). Let z = ε · f(x), where ε ∼ pα(ε) = logN (0, α2θ(x)), and assume pθ(z) = logN (µ, σ2). Then, we have
KL(pθ(z|x) ‖ p(z)) = 1 2σ2 ( α2(x) + µ2 ) − log α(x) σ − 1 2 . (4)
Substituting the expression for the KL divergence in eq. (3) inside eq. (2), and ignoring for simplicity the special case f(x) = 0, we obtain the following loss function for ReLU activations
L = 1 N N∑ i=1 Ez∼pθ(z|xi)[log p(yi|z)] + β logαθ(xi), (5)
and a similar expression for the Softplus. Notice that the first expectation can be approximated by sampling (in the experiments we use one single sample, as customary for dropout), and is just the average cross-entropy term that is typical in deep learning. The second term, which is new, penalizes the network for choosing a low variance for the noise, i.e. for letting more information pass through to the next layer. This loss can be optimized easily using stochastic gradient descent and the reparametrization trick of Kingma & Welling (2013) to back-propagate the gradient through the sampling operation.
5 VARIATIONAL AUTOENCODERS AND INFORMATION DROPOUT
In this section, we briefly outline a link between variational autoencoders (Kingma & Welling, 2013) and Information Dropout. A variational autoencoder (VAE) aims to reconstruct, given a training dataset D = {xi}, a latent random variable z such that the observed data x can be thought as being generated by the, usually simpler, variable z through some unknown generative process pθ(x|z). In practice, this is done by minimizing the negative variational lower-bound to the marginal log-likelihood of the data
L(θ) = 1 N N∑ i=1 Ez∼pθ(z|xi)[− log pθ(xi|z)] + KL(pθ(z|xi) ‖ p(z)),
which can be optimized easily using the SGVB method of Kingma & Welling (2013). Interestingly, when the task is reconstruction, that is when y = x, the IB loss function in eq. (2) reduces to
L(θ) = 1 N N∑ i=1 Ez∼pθ(z|xi)[− log pθ(xi|z)] + βKL(pθ(z|xi) ‖ p(z)).
Therefore, by letting β = 1 in the previous expression, we obtain exactly the loss function of a variational autoencoder, that is, the representation z computed by the Information Dropout layer coincides with the latent variable z computed by the VAE. This is in part to be expected, since the objective of Information Dropout is to create a representation of the data that is minimal sufficient for the task of reconstruction, and the latent variables of a VAE can be thought as such a representation. The term β in this case can be seen as managing the trade off between the fidelity of the reconstruction of the input from the representation (measured by the cross-entropy), against the compression factor (complexity) of the representation (measured by the KL-divergence).
6 EXPERIMENTS
We compare our method with the Dropout baseline on several standard benchmark datasets using different networks architecture, and highlight a few key properties of Information Dropout. All the models were implemented using TensorFlow (Abadi et al., 2015). As Kingma et al. (2015) also notice, letting the variance of the noise increase too much leads to poor local minima in the optimization process. To avoid this problem, we put the constraint α(x) < 0.7, so that the maximum variance of the log-normal error distribution will be approximatively 1, the same as binary dropout when using a drop probability of 0.5. In all experiments we divide the KL-divergence term by the number of training samples, so that for β = 1 the scaling of the KL-divergence term in similar to the one used by Variational Dropout (see Section 2).
Cluttered MNIST. To visually asses the ability of Information Dropout to create a representation of the input that is increasingly insensitive to nuisance factors, we train the All-CNN-96 network (Table 2) for classification on a Cluttered MNIST dataset (cf. Mnih et al., 2014), consisting of 96x96 images containing a single MNIST digit together with 21 distraction objects. The dataset is divided in 50.000 training images and 10.000 testing images. As shown in Figure 2, for small values of β, the network lets through both the objects of interest (digits) and distractors, to upper layers. By increasing the value of β, we force the network to disregard something, and the network decides to disregard the least discriminative components of the data, thereby building a better representation for the task. This behavior depends on the ability of Information Dropout to learn the structure of the nuisances in the dataset which, unlike other methods, is facilitated by the ability to select noise level on a per-sample basis.
Occluded CIFAR. Occlusions are a fundamental type of nuisance in vision for which it is difficult to hand-design invariant representations. To assess that the approximate minimal sufficient representation produced by Information Dropout has this invariance property, we created a new dataset by occluding images from CIFAR-10 with digits from MNIST (Figure 4a). We train the All-CNN-32 network (Table 2) to classify the CIFAR image. The information relative to the occluding MNIST digit is then a nuisance for the task, and therefore should be excluded from the final representation. To test this, we train a secondary network to classify the nuisance MNIST digit using only the the representation learned for the main task. When training with small values of β, the network has very little pressure to limit the presence of nuisance information in the representation, so we expect the nuisance classifier to perform better. On the other hand, increasing the value of β we expect the performance to degrade, since the representation will become increasingly minimal, and therefore invariant to nuisances. The results in Figure 4b confirm this intuition.
MNIST and CIFAR-10. Similar to Kingma et al. (2015), to see the effect of Information Dropout on different network sizes and architectures, we train on MNIST a network with 3 fully connected hidden layers with a variable number of hidden units, and we train on CIFAR-10 (Krizhevsky & Hinton, 2009) the All-CNN-32 convolutional network described in Table 2, using a variable percentage of all the filters. The fully connected network was trained for 80 epochs, using stochastic gradient descent with momentum with initial learning rate 0.07 and dropping the learning rate by 0.1 at 30 and 70 epochs. The CNN was trained for 160 epochs with initial learning rate 0.1 and dropping the learning rate by 0.1 at 80 and 120 epochs. We show the results in Figure 3. Information Dropout is comparable or outperforms binary dropout, especially on smaller networks. A possible explanation is
that dropout severely reduces the already limited capacity of the network, while Information Dropout can adapt the amount of noise to the data and to the size of the network so that the relevant information can still flow to the successive layers.
VAE. To validate what we said in Section 5, we replicate the basic variational autoencoder of Kingma & Welling (2013), implementing it both with Gaussian latent variables, as in the original, and with an Information Dropout layer. We trained both implementations for 300 epochs dropping the learning rate by 0.1 at 30 and 120 epochs. We report the results in the following table. The Information Dropout implementation has similar performance to the original, confirming that a variational autoencoder can be considered a special case of Information Dropout.
7 CONCLUSION
We introduced a new dropout method that can be seen as an efficient implementation of the Information Bottleneck principle and that helps the network learn the structure of the nuisance factors affecting the data and build representations that are insensitive to those nuisances, therefore improving generalization performance. We also analyzed from an information theoretic viewpoint the role that noise injected in a network has in learning nuisance invariance. Experimental evidence confirms the insights stemming from the theory thus developed.
Another interpretation of Information Dropout is as a way of biasing the network towards reconstructing representations of the data that are compatible with a Markov chain generative model,
making it more suited to data coming from hierarchical models, and in this sense is complementary to architectural constraint, such as convolutions, that instead bias the model toward geometrical tasks.
An important topic in representation learning, which we did not discuss explicitly, is whether we can also learn a “disentangled” representation of the data, and whether the factors of this representation are related to the latent factors of the real generative model. In Appendix B and Appendix C, we show that by adding independent (multiplicative) noise to the activations and by using the IB loss, we naturally favor representations which have mutually independent components and, in some restricted situations, we prove that we can disentangle the relevant part of the information from the nuisance variability. We leave proving more general results in this direction to a future work.
ACKNOWLEDGMENTS
Work supported by ARO, ONR, AFOSR.
A COMPUTATION OF THE KL-DIVERGENCE
Proposition (Information dropout cost for ReLU activations). Let z = ε · f(x), where ε ∼ pα(ε), and assume p(z) = qδ0(z) + c/z. Then, assuming f(x) 6= 0, we have
KL(pθ(z|x) ‖ p(z)) = −H(pα(x)(log ε)) + log(c)
In particular, if pα(ε) is chosen to be the log-normal distribution pα(ε) = logN (0, α2θ(x)), we have
KL(pθ(z|x) ‖ p(z)) = − logαθ(x) + const.
If instead f(x) = 0, we have KL(pθ(z|x) ‖ p(z)) = − log q.
Proof. If f(x) 6= 0, then we also have z 6= 0. Since the KL-divergence is invariant under parameter transformations we can write
KL(pθ(z|x) ‖ pθ(z)) = KL(pθ(log z|x) ‖ pθ(log z))
= ∫ log ( pθ(log z|x) pθ(log z) ) pθ(log z|x)dz
= ∫ log ( pα(x)(log ε) ) pα(x)(log ε)dε− log c
= −H(pα(x)(log ε))− log c.
For the second part, notice that by definition pα(x) = N (0, α2θ(x)) and
H(N (0, α)) = logαθ(x) + 1
2 log(2πe).
Finally, if f(x) = 0, then also z = 0, so p(z|x) = δ0(z). It is then easy to see that
KL(pθ(z|x) ‖ p(z)) = − log p(z = 0) = − log q.
Proposition (Information dropout cost for Softplus activations). Let z = ε·f(x), where ε ∼ pα(ε) = logN (0, α2θ(x)), and assume pθ(z) = logN (µ, σ2). Then, we have
KL(pθ(z|x) ‖ p(z)) = 1 2σ2 ( α2(x) + µ2 ) − log α(x) σ − 1 2 .
Proof. Since the KL divergence is invariant for reparametrizations, the divergence between two log-normal distributions is equal to the divergence between the corresponding normal distributions. Therefore, using the known formula for the KL divergence of normals, we get the desired result.
B EFFECTS OF USING A FACTORIZED APPROXIMATE PRIOR
In Information Dropout, we want to find a representation z ∼ q(z|x) that minimizes the objective
Hq(y|z) + βI(z;x).
This objective can be rewritten as
Hq(y|z) + βEx[KL(q(z|x) ‖ q(z))],
where the parameters of the posterior distribution q(z|x) are learned using a neural network. The prior distribution q(z) should in theory be computed from the posterior by integrating over x. However, since this is not feasible, we instead introduce a factorized approximation of the prior in the form p(z) = ∏ j pj(zj), and solve the problem
Hq(y|z) + βEx[KL(q(z|x) ‖ p(z))]. In this appendix, we show that the latter problem is indeed equivalent to the former when the components of the representation are mutually independent, and that assuming a factorized prior automatically favors the creation of such representations. In the following proposition, for simplicity, we concentrate on discrete random variables. Recall that the total correlation, or multivariate mutual information, of a variable z = (z1, . . . , zn) is defined as
TC(z) = ∑ j H(zj)−H(z)
= KL(q(z) ‖ ∏
j qj(zj)),
and that TC(z) = 0 if and only if the components of z are mutually independent. Proposition 3. Let z = (z1, . . . , zn) be a discrete random variable, let q(z|x) be a generic probability distribution, and let p(z) = ∏ i=1n p(zi) be a factorized prior distribution. Then, for any function F (q), a minimization problem in the form minimizeq,p F (q) + βEx[KL(q(z|x) ‖ p(z))],
is equivalent to minimizeq F (q) + β {Iq(z;x) + TCq(z)} , where Iq(z;x) is the mutual information and TCq(z) is the total correlation of z, assuming z ∼ q(z).
Proof. To prove the proposition, we just need to minimize with respect to p and substitute back the solution. Adding a Lagrange multiplier for the constrain ∑ zi pi(zi) = 1, the problem can be rewritten as
L(q, p) = F (q) + βEx [∑ z q(z|x) log q(z|x) p(z) dz ] + λ (∑ zi pi(zi)− 1 )
= F (q) + βEx [∑ z q(z|x) log q(z|x)∏n j=1 pj(zj) dz ] + λ (∑ zi pi(zi)− 1 ) .
Taking the derivative with respect to to pi(z̄i) we have
∂L(q, p) ∂pi(z̄i) = βEx [∑ z q(z|x) log q(z|x)∏n j=1 pj(zj) ] + λ
= −β ∑ zi=z̄i Ex[q(z|x)] pi(z̄i) + λ
= −β q(z̄i) p(z̄i) + λ.
Setting it to zero, we obtain p(zi) = q(zi), that is, the optimal factorized prior is the product of the marginals. Substituting it back in the second term (the only one containing p), we obtain
Ex[KL(q(z|x) ‖ p(z))] = Ex [∑ z q(z|x) log q(z|x)∏n j=1 qj(zj) ]
= Ex [∑ z q(z|x) ( log q(z|x) q(z) + log q(z)∏n j=1 qj(zj) )]
= Ex [∑ z q(z|x) log q(z|x) q(z) ] + ∑ z Ex[q(z|x)] log q(z)∏n j=1 qj(zj)
= Iq(z;x) + KL(q(z) ‖ ∏ j qj(zj))
= Iq(z;x) + TCq(z).
C INFORMATION BOTTLENECK PRINCIPLE AND DISENTANGLEMENT
In Appendix B, we showed that the loss function we use, in conjunction with a factorized prior, automatically favors representations of the data in which all the components are independent. In this appendix we show that, in the simple case of the data x coming from a Gaussian distribution, where our task is to recover one component of the data from a noisy observation, the solution of the Information Bottleneck Lagrangian when noise is added to the computation process naturally leads to recovering the independent components of the generative model. For simplicity, here we will use additive noise from a Gaussian distribution, but it is easily seen by exponentiating each random variable that this is exactly equivalent to using multiplicative noise from a log-normal distribution, which is the same case treated in the main paper.
Suppose we have a 2-dimensional Gaussian random variable
x = (x, y) ∼ N (0, I), and that our task is to recover y = eT2 x, given a noisy observation
x̂ = Ax+ ξ,
where A is an orthogonal matrix and ξ ∼ N (0, σ2ξ ) is some additive noise. We want to find an optimal representation of the input x̂ for the task y. To simplify the problem, we restrict to the representations in the form
z = Bx̂,
where B is an orthogonal matrix. Intuitively, the best representation for the task should be obtained by choosing B = A−1, since then we would have z = A−1x̂ = x+A−1ξ, and the component of x̂ relevant to the task, that is y, would be disentangled from the nuisance x. However, as we show in the following proposition, if we evaluate the representation using the conditional cross-entropy loss L = H(y|z), then all the choices of B are equivalent, while if we add noise to the representation and use the IB loss, we naturally obtain that the optimal representation is disentangled. Proposition 4. Let x ∼ N (0, I), x̂ = Ax+ ξ, where A is orthogonal and ξ ∼ N (0, σ2ξ ). We want to find a representation z of x for the task y = eT2 x.
(i) Assume the representation z is in the form z = Bx̂, where B is an orthogonal matrix. Then the cross-entropy loss L = H(y|z) does not depend on choice of B, so all the representations are equivalent for the cross-entropy loss.
(ii) Assume the representation z is in the form z = Bx̂+ , where B is an orthogonal matrix and the noise has distribution ∼ N (0,Σ = diag(σ21 , σ22)). Without loss of generality, assume that σ2 ≤ σ1. Then, for β small enough, the representation that minimizes the IB Lagrangian
L = H(y|z) + βI(x̂, z), is the disentangled representation obtained by choosing B = A−1 and, for β → 0+, we have σ1 → +∞ and σ2 → 0, so the added noise selectively preserve only information relative to the task.
Proof. For (i), notice that, since y = eT2 x = e T 2 (C −1z−Bξ) we have
y|z ∼ N (eT2 C−1z, σ2ξ ), therefore,
H(y|z) = log σξ + 1
2 log(2πe)
is independent from B.
For (ii), reasoning as before, we have
y|z ∼ N (eT2 C−1z, σ2ξ + eT2 CTΣCe2),
where we have defined C = BA. Since C is orthogonal, we can write Ce2 = (sin θ, cos θ) for some θ. Notice in particular that for θ = 0 we would have C = I , and so B = A−1. Using this, we can rewrite the expression above as
y|z ∼ N (eT2 C−1z, σ2ξ + σ21 sin2 θ + σ22 cos2 θ).
Therefore, the conditional entropy is
H(y|z) = log(σ2ξ + σ21 sin2 θ + σ22 cos2 θ) + const.
We now need to compute the mutual information I(x̂; z) term of the IB Lagrangian. Recall that the mutual information between two N-dimensional Gaussian variables x and z is
I(x; z) = 1
2 log
( |Σxx||Σzz|
|Σ|
) ,
where Σxx and Σzz are the covariances matrixes of x and z respectively and Σ is the covariance matrix of the pair (x, z). Using this, we obtain
I(x̂, z) = 1
2
[ log ( 1 +
1 + σ2ξ σ21
) + log ( 1 +
1 + σ2ξ σ22
)] .
Putting everything together, the IB loss function is
L = log(σ2ξ + σ21 cos2 θ + σ22 sin2 θ) + β
2
[ log ( 1 +
1 + σ2ξ σ21
) + log ( 1 +
1 + σ2ξ σ22
)] .
Since θ appears only in the first term, we can immediately minimize the loss with respect to θ. Recall that we are assuming σ2 ≤ σ1. Then, there are two possibilities: if σ2 < σ1, then we must have θ = 0 (or equivalently C = I and hence B = A−1). If instead σ1 = σ2, then all θ are equivalent. In both cases, the loss function simplifies to
L = log(σ2ξ + σ22) + β
2
[ log ( 1 +
1 + σ2ξ σ21
) + log ( 1 +
1 + σ2ξ σ22
)] .
The terms containing σ1 are now separate from those containing σ2 and we can minimize them independently. Doing so we obtain
σ1 → +∞, σ2 = f(β)
where f(β) is a function of β such that f(β)→ 0 as β → 0+. Finally, since we have now established that for β small enough we have σ2 < σ1, we can conclude from what we said before that we must have θ = 0, and therefore that B = A−1 and the representation is disentangled, as we wanted.
While the previous proposition deals with a very simple case, under restrictive hypotheses on the form of the representation, we conjecture that a similar property should hold more generally for any representation. For example, assume that we can find a mapping f(x) = (T (x), n), where T (x) is a sufficient statistic of the data for the task y, while n is independent from y. That is, f(x) can “disentangle” the component of the data relevant to the task from the nuisances. Then, it would be easy to minimize the IB Lagrangian by routing all the noise on n, while leaving T (x) untouched. We claim that the opposite should also happen: When the IB Lagrangian is minimized, the representation should be decomposed in a part relevant for the task, and a part which is independent. We leave further study of this problem to a future work.
D ADDITIONAL PLOTS

Paper summary
This paper develops a generalization of dropout using information theoretic
principles. The basic idea is that when learning a representation z of input x
with the aim of predicting y, we must choose a z such that it carries the least
amount of information about x, as long as it can predict y. This idea can be
formalized using the Information Bottleneck Lagrangian. This leads to an
optimization problem which is similar to the one derived for variational
dropout, the difference being that Information dropout allows for a scaling
factor associated with the KL divergence term that encourages noise. The amount
of noise being added is made a parameterized function of the data and this
function is optimized along with the rest of the model. Experimental results on
CIFAR-10 and MNIST show (small) improvements over binary dropout.
Strengths
- The paper highlights an important conceptual link between probabilistic
  variational methods and information theoretic methods, showing that dropout
can be generalized using both formalisms to arrive at very similar models.
- The presentation of the model is excellent.
- The experimental results on cluttered MNIST are impressive.
Weaknesses
- The results on CIFAR-10 in Figure 3(b) seem to be on a validation set (unless
  the axis label is a typo). It is not clear why the test set was not used. This
makes it hard to compare to results reported in Springenberg et al, as well as
other results in literature.
Quality
The theoretical exposition is high quality. Figure 2 gives a nice qualitative
assessment of what the model is doing. However, the experimental results
section can be made better, for example, by matching the results on CIFAR-10 as
reported in Springenberg et al. and trying to improve on those using information
dropout.
Clarity
The paper is well written and easy to follow.
Originality
The derivation of the information dropout optimization problem using IB
Lagrangian is novel. However, the final model is quite close to variational
dropout.
Significance
This paper will be of general interest to researchers in representation learning
because it highlights an alternative way to think about latent variables (as
information bottlenecks). However, unless the model can be shown to achieve
significant improvements over simple dropout, its wider impact is likely to be
limited.
Overall
The paper presents an insightful theoretical derivation and good preliminary
results. The experimental section can be improved.
Minor comments and suggestions -
- expecially -> especially
- trough -> through
- There is probably a minus sign missing in the expression for H(y|z) above Eq (2).
- Figure 3(a) has error bars, but 3(b) doesn't. It might be a good idea to have those
for Figure 3(b) as well.
- Please consider comparing Figure 2 with the activity map of a standard CNN
  trained with binary dropout, so we can see if similar filtering out is
happening there already.