NEURAL POPULATION INFOMAX
A framework is presented for unsupervised learning of representations based on infomax principle for large-scale neural populations. We use an asymptotic approximation to the Shannon’s mutual information for a large neural population to demonstrate that a good initial approximation to the global information-theoretic optimum can be obtained by a hierarchical infomax method. Starting from the initial solution, an efficient algorithm based on gradient descent of the final objective function is proposed to learn representations from the input datasets, and the method works for complete, overcomplete, and undercomplete bases. As confirmed by numerical experiments, our method is robust and highly efficient for extracting salient features from input datasets. Compared with the main existing methods, our algorithm has a distinct advantage in both the training speed and the robustness of unsupervised representation learning. Furthermore, the proposed method is easily extended to the supervised or unsupervised model for training deep structure networks.
A framework is presented for unsupervised learning of representations based on infomax principle for large-scale neural populations. We use an asymptotic approximation to the Shannon’s mutual information for a large neural population to demonstrate that a good initial approximation to the global information-theoretic optimum can be obtained by a hierarchical infomax method. Starting from the initial solution, an efficient algorithm based on gradient descent of the final objective function is proposed to learn representations from the input datasets, and the method works for complete, overcomplete, and undercomplete bases. As confirmed by numerical experiments, our method is robust and highly efficient for extracting salient features from input datasets. Compared with the main existing methods, our algorithm has a distinct advantage in both the training speed and the robustness of unsupervised representation learning. Furthermore, the proposed method is easily extended to the supervised or unsupervised model for training deep structure networks.
1 INTRODUCTION
How to discover the unknown structures in data is a key task for machine learning. Learning good representations from observed data is important because a clearer description may help reveal the underlying structures. Representation learning has drawn considerable attention in recent years (Bengio et al., 2013). One category of algorithms for unsupervised learning of representations is based on probabilistic models (Lewicki & Sejnowski, 2000; Hinton & Salakhutdinov, 2006; Lee et al., 2008), such as maximum likelihood (ML) estimation, maximum a posteriori (MAP) probability estimation, and related methods. Another category of algorithms is based on reconstruction error or generative criterion (Olshausen & Field, 1996; Aharon et al., 2006; Vincent et al., 2010; Mairal et al., 2010; Goodfellow et al., 2014), and the objective functions usually involve squared errors with additional constraints. Sometimes the reconstruction error or generative criterion may also have a probabilistic interpretation (Olshausen & Field, 1997; Vincent et al., 2010).
Shannon’s information theory is a powerful tool for description of stochastic systems and could be utilized to provide a characterization for good representations (Vincent et al., 2010). However, computational difficulties associated with Shannon’s mutual information (MI) (Shannon, 1948) have hindered its wider applications. The Monte Carlo (MC) sampling (Yarrow et al., 2012) is a convergent method for estimating MI with arbitrary accuracy, but its computational inefficiency makes it unsuitable for difficult optimization problems especially in the cases of high-dimensional input stimuli and large population networks. Bell and Sejnowski (Bell & Sejnowski, 1995; 1997) have directly applied the infomax approach (Linsker, 1988) to independent component analysis (ICA) of data with independent non-Gaussian components assuming additive noise, but their method requires that the number of outputs be equal to the number of inputs. The extensions of ICA to overcomplete or undercomplete bases incur increased algorithm complexity and difficulty in learning of parameters (Lewicki & Sejnowski, 2000; Kreutz-Delgado et al., 2003; Karklin & Simoncelli, 2011).
Since Shannon MI is closely related to ML and MAP (Huang & Zhang, 2017), the algorithms of representation learning based on probabilistic models should be amenable to information-theoretic treatment. Representation learning based on reconstruction error could be accommodated also by information theory, because the inverse of Fisher information (FI) is the Cramér-Rao lower bound on the mean square decoding error of any unbiased decoder (Rao, 1945). Hence minimizing the reconstruction error potentially maximizes a lower bound on the MI (Vincent et al., 2010).
Related problems arise also in neuroscience. It has long been suggested that the real nervous systems might approach an information-theoretic optimum for neural coding and computation (Barlow, 1961; Atick, 1992; Borst & Theunissen, 1999). However, in the cerebral cortex, the number of neurons is huge, with about 105 neurons under a square millimeter of cortical surface (Carlo & Stevens, 2013). It has often been computationally intractable to precisely characterize information coding and processing in large neural populations.
To address all these issues, we present a framework for unsupervised learning of representations in a large-scale nonlinear feedforward model based on infomax principle with realistic biological constraints such as neuron models with Poisson spikes. First we adopt an objective function based on an asymptotic formula in the large population limit for the MI between the stimuli and the neural population responses (Huang & Zhang, 2017). Since the objective function is usually nonconvex, choosing a good initial value is very important for its optimization. Starting from an initial value, we use a hierarchical infomax approach to quickly find a tentative global optimal solution for each layer by analytic methods. Finally, a fast convergence learning rule is used for optimizing the final objective function based on the tentative optimal solution. Our algorithm is robust and can learn complete, overcomplete or undercomplete basis vectors quickly from different datasets. Experimental results showed that the convergence rate of our method was significantly faster than other existing methods, often by an order of magnitude. More importantly, the number of output units processed by our method can be very large, much larger than the number of inputs. As far as we know, no existing model can easily deal with this situation.
2 METHODS
2.1 APPROXIMATION OF MUTUAL INFORMATION FOR NEURAL POPULATIONS
Suppose the input x is a K-dimensional vector, x = (x1, · · · , xK)T , the outputs of N neurons are denoted by a vector, r = (r1, · · · , rN )T , where we assume N is large, generally N K. We denote random variables by upper case letters, e.g., random variables X and R, in contrast to their vector values x and r. The MI between X and R is defined by I(X;R) = 〈 ln p(x|r)p(x) 〉 r,x , where 〈·〉r,x denotes the expectation with respect to the probability density function (PDF) p(r,x).
Our goal is to maxmize MI I(X;R) by finding the optimal PDF p(r|x) under some constraint conditions, assuming that p(r|x) is characterized by a noise model and activation functions f(x;θn) with parameters θn for the n-th neuron (n = 1, · · · , N ). In other words, we optimize p(r|x) by solving for the optimal parameters θn. Unfortunately, it is intractable in most cases to solve for the optimal parameters that maximizes I(X;R). However, if p(x) and p(r|x) are twice continuously differentiable for almost every x ∈ RK , then for large N we can use an asymptotic formula to approximate the true value of I(X;R) with high accuracy (Huang & Zhang, 2017):
I(X;R) ' IG = 1
2
〈 ln ( det ( G(x)
2πe ))〉 x +H(X), (1)
where det (·) denotes the matrix determinant and H(X) = −〈ln p(x)〉x is the stimulus entropy, G(x) = J(x) + P (x) , (2)
J(x) = − 〈 ∂2 ln p (r|x) ∂x∂xT 〉 r|x , (3)
P(x) = −∂ 2 ln p (x)
∂x∂xT . (4) Assuming independent noises in neuronal responses, we have p(r|x) = ∏N n=1 p(rn|x;θn),
and the Fisher information matrix becomes J(x) ≈ N ∑K1 k=1 αkS(x;θk), where S(x;θk) =
〈 ∂ ln p(r|x;θk)
∂x ∂ ln p(r|x;θk) ∂xT 〉 r|x and αk > 0 (k = 1, · · · ,K1) is the population density of param-
eter θk, with ∑K1 k=1 αk = 1, and 1 ≤ K1 ≤ N (see Appendix A.1 for details). Since the cerebral cortex usually forms functional column structures and each column is composed of neurons with the same properties (Hubel & Wiesel, 1962), the positive integer K1 can be regarded as the number of distinct classes in the neural population.
Therefore, given the activation function f(x;θk), our goal becomes to find the optimal population distribution density αk of parameter vector θk so that the MI between the stimulus x and the response r is maximized. By Eq. (1), our optimization problem can be stated as follows:
minimize QG[{αk}] = − 1
2 〈ln (det (G(x)))〉x , (5)
subject to K1∑ k=1 αk = 1, αk > 0, ∀k = 1, · · · ,K1. (6)
Since QG[{αk}] is a convex function of {αk} (Huang & Zhang, 2017), we can readily find the optimal solution for small K by efficient numerical methods. For large K, however, finding an optimal solution by numerical methods becomes intractable. In the following we will propose an alternative approach to this problem. Instead of directly solving for the density distribution {αk}, we optimize the parameters {αk} and {θk} simultaneously under a hierarchical infomax framework.
2.2 HIERARCHICAL INFOMAX
For clarity, we consider neuron model with Poisson spikes although our method is easily applicable to other noise models. The activation function f(x;θn) is generally a nonlinear function, such as sigmoid and rectified linear unit (ReLU) (Nair & Hinton, 2010). We assume that the nonlinear function for the n-th neuron has the following form: f(x;θn) = f̃(yn; θ̃n), where
yn = w T nx. (7)
with wn being aK-dimensional weights vector, f̃(yn; θ̃n) is a nonlinear function, θn = (wTn , θ̃ T n ) T and θ̃n are the parameter vectors (n = 1, · · · , N ). In general, it is very difficult to find the optimal parameters, θn, n = 1, · · · , N , for the following reasons. First, the number of output neuronsN is very large, usuallyN K. Second, the activation function f(x;θn) is a nonlinear function, which usually leads to a nonconvex optimization problem. For nonconvex optimization problems, the selection of initial values often has a great influence on the final optimization results. Our approach meets these challenges by making better use of the large number of neurons and by finding good initial values by a hierarchical infomax method.
We divide the nonlinear transformation into two stages, mapping first from x to yn (n = 1, · · · , N ), and then from yn to f̃(yn; θ̃n), where yn can be regarded as the membrane potential of the n-th neuron, and f̃(yn; θ̃n) as its firing rate. As with the real neurons, we assume that the membrane potential is corrupted by noise:
Y̆n = Yn + Zn, (8) where Zn ∼ N ( 0,σ2 ) is a normal distribution with mean 0 and variance σ2. Then the mean membrane potential of the k-th class subpopulation with Nk = Nαk neurons is given by
Ȳk = 1
Nk Nk∑ n=1 Y̆kn = Yk + Z̄k, k = 1, · · · ,K1, (9)
Z̄k ∼ N (0, N−1k σ 2). (10)
Define vectors y̆ = (y̆1, · · · , y̆N )T , ȳ = (ȳ1, · · · , ȳK1)T and y = (y1, · · · , yK1)T , where yk = wTk x (k = 1, · · · ,K1). Notice that y̆n (n = 1, · · · , N ) is also divided into K1 classes, the same as for rn. If we assume f(x;θk) = f̃(ȳk; θ̃k), i.e. assuming an additive Gaussian noise for yn (see Eq. 9), then the random variables X , Y , Y̆ , Ȳ and R form a Markov chain, denoted by X → Y → Y̆ → Ȳ → R (see Figure 1), and we have the following proposition (see Appendix A.2).
Proposition 1. With the random variables X , Y , Y̆ , Ȳ , R and Markov chain X → Y → Y̆ → Ȳ → R, the following equations hold,
I(X;R) = I(Y ;R) ≤ I(Y̆ ;R) ≤ I(Ȳ ;R), (11) I(X;R) ≤ I(X; Ȳ ) = I(X; Y̆ ) ≤ I(X;Y ), (12)
and for large Nk (k = 1, · · · ,K1),
I(Y̆ ;R) ' I(Ȳ ;R) ' I(Y ;R) = I(X;R), (13) I(X;Y ) ' I(X; Ȳ ) = I(X; Y̆ ). (14)
A major advantage of incorporating membrane noise is that it facilitates finding the optimal solution by using the infomax principle. Moreover, the optimal solution obtained this way is more robust; that is, it discourages overfitting and has a strong ability to resist distortion. With vanishing noise σ2 → 0, we have Ȳk → Yk, f̃(ȳk; θ̃k) ' f̃(yk; θ̃k) = f(x;θk), so that Eqs. (13) and (14) hold as in the case of large Nk.
To optimize MI I(Y ;R), the probability distribution of random variable Y , p(y), needs to be determined, i.e. maximizing I(Y ;R) about p(y) under some constraints should yield an optimal distribution: p∗(y) = arg maxp(y) I(Y ;R). Let C = maxp(y) I (Y ;R) be the channel capacity of neural population coding, and we always have I(X;R) ≤ C (Huang & Zhang, 2017). To find a suitable linear transformation from X to Y that is compatible with this distribution p∗(y), a reasonable choice is to maximize I(X; Y̆ ) (≤ I(X;Y )), where Y̆ is a noise-corrupted version of Y . This implies minimum information loss in the first transformation step. However, there may exist many transformations from X to Y̆ that maximize I(X; Y̆ ) (see Appendix A.3.1). Ideally, if we can find a transformation that maximizes both I(X; Y̆ ) and I(Y ;R) simultaneously, then I(X;R) reaches its maximum value: I(X;R) = maxp(y) I (Y ;R) = C.
From the discussion above we see that maximizing I(X;R) can be divided into two steps, namely, maximizing I(X; Y̆ ) and maximizing I(Y ;R). The optimal solutions of max I(X; Y̆ ) and max I(Y ;R) will provide a good initial approximation that tend to be very close to the optimal solution of max I(X;R).
Similarly, we can extend this method to multilayer neural population networks. For example, a twolayer network with outputs R(1) and R(2) form a Markov chain, X → R̃(1) → R(1) → R̄(1) →
R(2), where random variable R̃(1) is similar to Y , random variable R(1) is similar to Y̆ , and R̄(1) is similar to Ȳ in the above. Then we can show that the optimal solution of max I(X;R(2)) can be approximated by the solutions of max I(X;R(1)) and max I(R̃(1);R(2)), with I(R̃(1);R(2)) ' I(R̄(1);R(2)).
More generally, consider a highly nonlinear feedforward neural network that maps the input x to output z, with z = F (x;θ) = hL ◦ · · · ◦ h1 (x), where hl (l = 1, · · · , L) is a linear or nonlinear function (Montufar et al., 2014). We aim to find the optimal parameter θ by maximizing I (X;Z). It is usually difficult to solve the optimization problem when there are many local extrema for F (x;θ). However, if each function hl is easy to optimize, then we can use the hierarchical infomax method described above to get a good initial approximation to its global optimization solution, and go from there to find the final optimal solution. This information-theoretic consideration from the neural population coding point of view may help explain why deep structure networks with unsupervised pre-training have a powerful ability for learning representations.
2.3 THE OBJECTIVE FUNCTION
The optimization processes for maximizing I(X; Y̆ ) and maximizing I(Y ;R) are discussed in detail in Appendix A.3. First, by maximizing I(X; Y̆ ) (see Appendix A.3.1 for details), we can get the optimal weight parameter wk (k = 1, · · · ,K1, see Eq. 7) and its population density αk (see Eq. 6) which satisfy
W = [w1, · · · ,wK1 ] = aU0Σ −1/2 0 C, (15) α1 = · · · = αK1 = K−11 , (16)
where a = √ K1K −1 0 , C = [c1, · · · , cK1 ] ∈ RK0×K1 , CC
T = IK0 , IK0 is a K0 × K0 identity matrix with integer K0 ∈ [1,K], the diagonal matrix Σ0 ∈ RK0×K0 and matrix U0 ∈ RK×K0 are given in (A.44) and (A.45), with K0 given by Eq. (A.52). Matrices Σ0 and U0 can be obtained by Σ and U with UT0 U0 = IK0 and U0Σ0U T 0 ≈ UΣU T ≈ 〈 xxT 〉 x
(see Eq. A.23). The optimal weight parameter wk (15) means that the input variable x must first undergo a whiteninglike transformation x̂ = Σ−1/20 U T 0 x, and then goes through the transformation y = aC
T x̂, with matrix C to be optimized below. Note that weight matrix W satisfies rank(W) = min(K0,K1), which is a low rank matrix, and its low dimensionality helps reduce overfitting during training (see Appendix A.3.1).
By maximizing I (Y ;R) (see Appendix A.3.2), we further solve the the optimal parameters θ̃k for the nonlinear functions f̃(yk; θ̃k), k = 1, · · · ,K1. Finally, the objective function for our optimization problem (Eqs. 5 and 6) turns into (see Appendix A.3.3 for details):
minimize Q [C] =− 1 2
〈 ln ( det ( CΦ̂C T ))〉
x̂ , (17)
subject to CCT = IK0 , (18) where Φ̂ = diag ( φ(ŷ1) 2, · · · , φ(ŷK1)2 ) , φ(ŷk) = a−1 |∂gk(ŷk)/∂ŷk| (k = 1, · · · ,K1), gk(ŷk) =
2 √ f̃(ŷk; θ̃k), ŷk = a−1yk = cTk x̂, and x̂ = Σ −1/2 0 U T 0 x. We apply the gradient descent method to optimize the objective function, with the gradient of Q[C] given by:
dQ[C]
dC = −
〈( CΦ̂C T )−1 CΦ̂ + x̂ωT 〉
x̂
, (19)
where ω = (ω1, · · · , ωK1) T , ωk = φ(ŷk)φ′(ŷk)cTk
( CΦ̂C T )−1 ck, k = 1, · · · ,K1.
When K0 = K1 (or K0 > K1), the objective function Q[C] can be reduced to a simpler form, and its gradient is also easy to compute (see Appendix A.4.1). However, when K0 < K1, it is computationally expensive to update C by applying the gradient of Q[C] directly, since it requires matrix inversion for every x̂. We use another objective function Q̂[C] (see Eq. A.118) which is an approximation to Q[C], but its gradient is easier to compute (see Appendix A.4.2). The function
Q̂[C] is the approximation of Q[C], ideally they have the same optimal solution for the parameter C.
Usually, for optimizing the objective in Eq. 17, the orthogonality constraint (Eq. 18) is unnecessary. However, this orthogonality constraint can accelerate the convergence rate if we employ it for the initial iteration to update C (see Appendix A.5).
3 EXPERIMENTAL RESULTS
We have applied our methods to the natural images from Olshausen’s image dataset (Olshausen & Field, 1996) and the images of handwritten digits from MNIST dataset (LeCun et al., 1998) using Matlab 2016a on a computer with 12 Intel CPU cores (2.4 GHz). The gray level of each raw image was normalized to the range of 0 to 1. M image patches with size w × w = K for training were randomly sampled from the images. We used the Poisson neuron model with a modified sigmoidal
tuning function f̃(y; θ̃) = 1 4(1+exp(−βy−b))2 , with g(y) = 2 √ f̃(y; θ̃) = 11+exp(−βy−b) , where
θ̃ = (β, b) T . We obtained the initial values (see Appendix A.3.2): b0 = 0 and β0 ≈ 1.81 √ K1K −1 0 . For our experiments, we set β = 0.5β0 for iteration epoch t = 1, · · · , t0 and β = β0 for t = t0 + 1, · · · , tmax, where t0 = 50. Firstly, we tested the case of K = K0 = K1 = 144 and randomly sampled M = 105 image patches with size 12×12 from the Olshausen’s natural images, assuming thatN = 106 neurons were divided into K1 = 144 classes and = 1 (see Eq. A.52 in Appendix). The input patches were preprocessed by the ZCA whitening filters (see Eq. A.68). To test our algorithms, we chose the batch size to be equal to the number of training samples M , although we could also choose a smaller batch size. We updated the matrix C from a random start, and set parameters tmax = 300, v1 = 0.4, and τ = 0.8 for all experiments.
In this case, the optimal solution C looked similar to the optimal solution of IICA (Bell & Sejnowski, 1997). We also compared with the fast ICA algorithm (FICA) (Hyvärinen, 1999), which is faster than IICA. We also tested the restricted Boltzmann machine (RBM) (Hinton et al., 2006) for a unsupervised learning of representations, and found that it could not easily learn Gabor-like filters from Olshausen’s image dataset as trained by contrastive divergence. However, an improved method by adding a sparsity constraint on the output units, e.g., sparse RBM (SRBM) (Lee et al., 2008) or sparse autoencoder (Hinton, 2010), could attain Gabor-like filters from this dataset. Similar results with Gabor-like filters were also reproduced by the denoising autoencoders (Vincent et al., 2010), which method requires a careful choice of parameters, such as noise level, learning rate, and batch size.
In order to compare our methods, i.e. Algorithm 1 (Alg.1, see Appendix A.4.1) and Algorithm 2 (Alg.2, see Appendix A.4.2), with other methods, i.e. IICA, FICA and SRBM, we implemented these algorithms using the same initial weights and the same training data set (i.e. 105 image patches preprocessed by the ZCA whitening filters). To get a good result by IICA, we must carefully select the parameters; we set the batch size as 50, the initial learning rate as 0.01, and final learning rate as 0.0001, with an exponential decay with the epoch of iterations. IICA tends to have a faster convergence rate for a bigger batch size but it may become harder to escape local minima. For FICA, we chose the nonlinearity function f(u) = log cosh(u) as contrast function, and for SRBM, we set the sparseness control constant p as 0.01 and 0.03. The number of epoches for iterations was set to 300 for all algorithms. Figure 2 shows the filters learned by our methods and other methods. Each filter in Figure 2(a) corresponds to a column vector of matrix Č (see Eq. A.69), where each vector for display is normalized by čk ← čk/max(|č1,k|, · · · , |čK,k|), k = 1, · · · ,K1. The results in Figures 2(a), 2(b) and 2(c) look very similar to one another, and slightly different from the results in Figure 2(d) and 2(e). There are no Gabor-like filters in Figure 2(f), which corresponds to SRBM with p = 0.03.
Figure 3 shows how the coefficient entropy (CFE) (see Eq. A.122) and the conditional entropy (CDE) (see Eq. A.125) varied with training time. We calculated CFE and CDE by sampling once every 10 epoches from a total of 300 epoches. These results show that our algorithms had a fast convergence rate towards stable solutions while having CFE and CDE values similar to the algorithm of IICA, which converged much more slowly. Here the values of CFE and CDE should be as small
100 101 102
1.8
1.85
1.9
1.95
2
co ef
fic ie
nt e
nt ro
py (
bi ts ) Alg.1 Alg.2 IICA FICA SRBM (p = 0.01) SRBM (p = 0.03)
100 101 102
-400
-350
-300
-250
-200
-150
co nd
iti on
al e
nt ro
py (
bi ts ) Alg.1 Alg.2 IICA
100 101 102
-200
-100
0
100
200
300
co nd
iti on
al e
nt ro
py (
bi ts
)
SRBM (p = 0.01) SRBM (p = 0.03) SRBM (p = 0.05) SRBM (p = 0.10)
as possible for a good representation learned from the same data set. Here we set epoch number t0 = 50 in our algorithms (see Alg.1 and Alg.2), and the start time was set to 1 second. This explains the step seen in Figure 3 (b) for Alg.1 and Alg.2 since the parameter β was updated when epoch number t = t0. FICA had a convergence rate close to our algorithms but had a big CFE, which is reflected by the quality of the filter results in Figure 2. The convergence rate and CFE for SRBM were close to IICA, but SRBM had a much bigger CDE than IICA, which implies that the information had a greater loss when passing through the system optimized by SRBM than by IICA or our methods.
From Figure 3(c) we see that the CDE (or MI I(X;R), see Eq. A.124 and A.125) decreases (or increases) with the increase of the value of the sparseness control constant p. Note that a smaller p means sparser outputs. Hence, in this sense, increasing sparsity may result in sacrificing some information. On the other hand, a weak sparsity constraint may lead to failure of learning Gaborlike filters (see Figure 2(f)), and increasing sparsity has an advantage in reducing the impact of noise in many practical cases. Similar situation also occurs in sparse coding (Olshausen & Field, 1997), which provides a class of algorithms for learning overcomplete dictionary representations of the input signals. However, its training is time consuming due to its expensive computational cost, although many new training algorithms have emerged (e.g. Aharon et al., 2006; Elad & Aharon, 2006; Lee et al., 2006; Mairal et al., 2010). See Appendix A.5 for additional experimental results.
4 CONCLUSIONS
In this paper, we have presented a framework for unsupervised learning of representations via information maximization for neural populations. Information theory is a powerful tool for machine learning and it also provides a benchmark of optimization principle for neural information processing in nervous systems. Our framework is based on an asymptotic approximation to MI for a large-scale neural population. To optimize the infomax objective, we first use hierarchical infomax to obtain a good approximation to the global optimal solution. Analytical solutions of the hierarchical infomax are further improved by a fast convergence algorithm based on gradient descent. This method allows us to optimize highly nonlinear neural networks via hierarchical optimization using infomax principle.
From the viewpoint of information theory, the unsupervised pre-training for deep learning (Hinton & Salakhutdinov, 2006; Bengio et al., 2007) may be reinterpreted as a process of hierarchical infomax, which might help explain why unsupervised pre-training helps deep learning (Erhan et al., 2010). In our framework, a pre-whitening step can emerge naturally by the hierarchical infomax, which might also explain why a pre-whitening step is useful for training in many learning algorithms (Coates et al., 2011; Bengio, 2012).
Our model naturally incorporates a considerable degree of biological realism. It allows the optimization of a large-scale neural population with noisy spiking neurons while taking into account of multiple biological constraints, such as membrane noise, limited energy, and bounded connection weights. We employ a technique to attain a low-rank weight matrix for optimization, so as to reduce the influence of noise and discourage overfitting during training. In our model, many parameters are optimized, including the population density of parameters, filter weight vectors, and parameters for nonlinear tuning functions. Optimizing all these model parameters could not be easily done by many other methods.
Our experimental results suggest that our method for unsupervised learning of representations has obvious advantages in its training speed and robustness over the main existing methods. Our model has a nonlinear feedforward structure and is convenient for fast learning and inference. This simple and flexible framework for unsupervised learning of presentations should be readily extended to training deep structure networks. In future work, it would interesting to use our method to train deep structure networks with either unsupervised or supervised learning.
ACKNOWLEDGMENTS
We thank Prof. Honglak Lee for sharing Matlab code for algorithm comparison, Prof. Shan Tan for discussions and comments and Kai Liu for helping draw Figure 1. Supported by grant NIH-NIDCD R01 DC013698.
APPENDIX
A.1 FORMULAS FOR APPROXIMATION OF MUTUAL INFORMATION
It follows from I(X;R) = 〈
ln p(x|r)p(x) 〉 r,x and Eq. (1) that the conditional entropy should read:
H(X|R) = −〈ln p(x|r)〉r,x ' − 1
2
〈 ln ( det ( G(x)
2πe ))〉 x . (A.1)
The Fisher information matrix J(x) (see Eq. 3), which is symmetric and positive semidefinite, can be written also as
J(x) =
〈 ∂ ln p(r|x)
∂x
∂ ln p(r|x) ∂xT 〉 r|x . (A.2)
If we suppose p(r|x) is conditional independent, namely, p(r|x) = ∏N n=1 p(rn|x;θn), then we have (see Huang & Zhang, 2017)
J(x) = N ∫ Θ p(θ)S(x;θ)dθ, (A.3)
S(x;θ) =
〈 ∂ ln p(r|x;θ)
∂x
∂ ln p(r|x;θ) ∂xT 〉 r|x , (A.4)
where p(θ) is the population density function of parameter θ,
p(θ) = 1
N N∑ n=1 δ(θ − θn), (A.5)
and δ(·) denotes the Dirac delta function. It can be proved that the approximation function of MI IG[p(θ)] (Eq. 1) is concave about p(θ) (Huang & Zhang, 2017). In Eq. (A.3), we can approximate the continuous integral by a discrete summation for numerical computation,
J(x) ≈ N K1∑ k=1 αkS(x;θk), (A.6)
where ∑K1 k=1 αk = 1, αk > 0, k = 1, · · · ,K1, 1 ≤ K1 ≤ N .
For Poisson neuron model, by Eq. (A.4) we have (see Huang & Zhang, 2017)
p(r|x;θ) = f(x;θ) r
r! exp (−f(x;θ)), (A.7)
S(x;θ) = 1
f(x;θ)
∂f(x;θ)
∂x
∂f(x;θ)
∂xT
= ∂g(x;θ)
∂x
∂g(x;θ)
∂xT , (A.8)
where f(x;θ) ≥ 0 is the activation function (mean response) of neuron and g(x;θ) = 2 √ f(x;θ). (A.9)
Similarly, for Gaussian noise model, we have
p(r|x;θ) = 1 σ √ 2π exp
( − (r − f(x;θ)) 2
2σ2
) , (A.10)
S(x;θ) = 1 σ2 ∂f(x;θ) ∂x ∂f(x;θ) ∂xT , (A.11)
where σ > 0 denotes the standard deviation of noise.
Sometimes we do not know the specific form of p(x) and only know M samples, x1, · · · , xM , which are independent and identically distributed (i.i.d.) samples drawn from the distribution p(x). Then we can use the empirical average to approximate the integral in Eq. (1):
IG ≈ 1
2 M∑ m=1 ln (det (G(xm))) +H (X) . (A.12)
A.2 PROOF OF PROPOSITION 1
Proof. It follows from the data-processing inequality (Cover & Thomas, 2006) that
I(X;R) ≤ I(Y ;R) ≤ I(Y̆ ;R) ≤ I(Ȳ ;R), (A.13) I(X;R) ≤ I(X; Ȳ ) ≤ I(X; Y̆ ) ≤ I(X;Y ). (A.14)
Since p(ȳk|x) = p(y̆k1 , · · · , y̆kNk |x) = N (w T k x, N −1 k σ 2), k = 1, · · · ,K1, (A.15)
we have
p(ȳ|x) = p(y̆|x), (A.16) p(ȳ) = p(y̆), (A.17)
I(X; Ȳ ) = I(X; Y̆ ). (A.18)
Hence, by (A.14) and (A.18), expression (12) holds. On the other hand, when Nk is large, from Eq. (10) we know that the distribution of Z̄k, namely, N ( 0,N−1k σ 2 ) , approaches a Dirac delta function δ(z̄k). Then by (7) and (9) we have p (r|ȳ) ' p(r|y) = p (r|x) and
I(X;R) = I (Y ;R)− 〈
ln p (r|y) p (r|x) 〉 r,x = I (Y ;R) , (A.19)
I (Y ;R) = I(Ȳ ;R)− 〈
ln p (r|ȳ) p (r|y) 〉 r,y,ȳ ' I(Ȳ ;R), (A.20)
I (Y ;R) = I(Y̆ ;R)− 〈
ln p (r|y̆) p (r|y) 〉 r,y,y̆ ' I(Y̆ ;R), (A.21)
I(X;Y ) = I(X; Ȳ )− 〈
ln p (x|ȳ) p (x|y) 〉 x,y,ȳ ' I(X; Ȳ ). (A.22)
It follows from (A.13) and (A.19) that (11) holds. Combining (11), (12) and (A.20)–(A.22), we immediately get (13) and (14). This completes the proof of Proposition 1.
A.3 HIERARCHICAL OPTIMIZATION FOR MAXIMIZING I(X;R)
In the following, we will discuss the optimization procedure for maximizing I(X;R) in two stages: maximizing I(X; Y̆ ) and maximizing I(Y ;R).
A.3.1 THE 1ST STAGE
In the first stage, our goal is to maximize the MI I(X; Y̆ ) and get the optimal parameters wk (k = 1, · · · ,K1). Assume that the stimulus x has zero mean (if not, let x ← x − 〈x〉x) and covariance matrix Σx. It follows from eigendecomposition that
Σx = 〈 xxT 〉 x ≈ 1 M − 1 XXT = UΣUT , (A.23)
where X = [x1, · · · , xM ], U = [u1, · · · ,uK ] ∈ RK×K is an unitary orthogonal matrix and Σ = diag ( σ21 , · · · , σ2K ) is a positive diagonal matrix with σ1 ≥ · · · ≥ σK > 0. Define
x̃ = Σ−1/2UTx, (A.24)
w̃k = Σ 1/2UTwk, (A.25)
yk = w̃ T k x̃, (A.26)
where k = 1, · · · ,K1. The covariance matrix of x̃ is given by Σx̃ = 〈 x̃x̃T 〉 x̃ ≈ IK , (A.27)
and IK is a K ×K identity matrix. From (1) and (A.11) we have I(X; Y̆ ) = I(X̃; Y̆ ) and
I(X̃; Y̆ ) ' I ′G = 1
2 ln
( det ( G̃
2πe
)) +H(X̃), (A.28)
G̃ ≈ Nσ−2 K1∑ k=1 αkw̃kw̃ T k + IK . (A.29)
The following approximations are useful (see Huang & Zhang, 2017):
p(x̃) ≈ N (0, IK), (A.30)
P(x̃) = −∂ 2 ln p (x̃)
∂x̃∂x̃T ≈ IK . (A.31)
By the central limit theorem, the distribution of random variable X̃ is closer to a normal distribution than the distribution of the original random variable X . On the other hand, the PCA models assume multivariate gaussian data whereas the ICA models assume multivariate non-gaussian data. Hence by a PCA-like whitening transformation (A.24) we can use the approximation (A.31) with the Laplace’s method of asymptotic expansion, which only requires that the peak be close to its mean while random variable X̃ needs not be exactly Gaussian.
Without any constraints on the Gaussian channel of neural populations, especially the peak firing rates, the capacity of this channel may grow indefinitely: I(X̃; Y̆ ) → ∞. The most common constraint on the neural populations is an energy or power constraint which can also be regarded as a signal-to-noise ratio (SNR) constraint. The SNR for the output y̆n of the n-th neuron is given by
SNRn = 1
σ2
〈( wTnx )2〉 x ≈ 1 σ2 w̃Tn w̃n, n = 1, · · · , N . (A.32)
We require that 1
N N∑ n=1 SNRn ≈ 1 σ2 K1∑ k=1 αkw̃ T k w̃k ≤ ρ, (A.33)
where ρ is a positive constant. Then by Eq. (A.28), (A.29) and (A.33), we have the following optimization problem:
minimize Q′G[Ŵ] = − 1 2 ln ( det ( Nσ−2ŴŴ T + IK )) , (A.34)
subject to h = Tr ( ŴŴ T ) − E ≤ 0, (A.35)
where Tr (·) denotes matrix trace and
Ŵ = W̃A 1/2
= Σ1/2UTWA1/2 = [ŵ1, · · · , ŵK1 ], (A.36) A = diag (α1, · · · , αK1), (A.37) W = [w1, · · · ,wK1 ], (A.38) W̃ = [w̃1, · · · , w̃K1 ], (A.39) E = ρσ2. (A.40)
Here E is a constant that does not affect the final optimal solution so we set E = 1. Then we obtain an optimal solution as follows:
W = aU0Σ −1/2 0 V T 0 , (A.41)
A = K−11 IK1 , (A.42)
a = √ EK1K −1 0 = √ K1K −1 0 , (A.43)
Σ0 = diag ( σ21 , · · · , σ2K0 ) , (A.44) U0 = U (:, 1:K0) ∈ RK×K0 , (A.45) V0 = V (:, 1:K0) ∈ RK1×K0 , (A.46)
where V = [v1, · · · ,vK1 ] is an K1 ×K1 unitary orthogonal matrix, parameter K0 represents the size of the reduced dimension (1 ≤ K0 ≤ K), and its value will be determined below. Now the optimal parameters wn (n = 1, · · · , N ) are clustered into K1 classes (see Eq. A.6) and obey an uniform discrete distribution (see also Eq. A.60 in Appendix A.3.2).
When K = K0 = K1, the optimal solution of W in Eq. (A.41) is a whitening-like filter. When V = IK , the optimal matrix W is the principal component analysis (PCA) whitening filters. In the symmetrical case with V = U, the optimal matrix W becomes a zero component analysis (ZCA) whitening filter. IfK < K1, this case leads to an overcomplete solution, whereas whenK0 ≤ K1 < K, the undercomplete solution arises. Since K0 ≤ K1 and K0 ≤ K, Q′G achieves its minimum when K0 = K. However, in practice other factors may prevent it from reaching this minimum. For example, consider the average of squared weights,
ς = K1∑ k=1 αk ‖wk‖2 = Tr ( WAWT ) = E K0 K0∑ k=1 σ−2k , (A.47)
where ‖·‖ denotes the Frobenius norm. The value of ς is extremely large when any σk becomes vanishingly small. For real neurons these weights of connection are not allowed to be too large. Hence we impose a limitation on the weights: ς ≤ E1, where E1 is a positive constant. This yields another constraint on the objective function,
h̃ = E
K0 K0∑ k=1 σ−2k − E1 ≤ 0. (A.48)
From (A.35) and (A.48) we get the optimal K0 = arg maxK̃0 ( EK̃−10 ∑K̃0 k=1 σ −2 k ) . By this constraint, small values of σ2k will often result in K0 < K and a low-rank matrix W (Eq. A.41).
On the other hand, the low-rank matrix W can filter out the noise of stimulus x. Consider the transformation Y = WTX with X = [x1, · · · , xM ] and Y = [y1, · · · , yM ] for M samples. It follows from the singular value decomposition (SVD) of X that
X = USṼ T , (A.49)
where U is given in (A.23), Ṽ is aM×M unitary orthogonal matrix, S is aK×M diagonal matrix with non-negative real numbers on the diagonal, Sk,k = √ M − 1σk (k = 1, · · · ,K, K ≤M ), and SST = (M − 1)Σ. Let X̆ = √ M − 1U0Σ1/20 ṼT0 ≈ X, (A.50)
where Ṽ0 = Ṽ (:, 1:K0) ∈ RM×K0 , Σ0 and U0 are given in (A.44) and (A.45), respectively. Then
Y = WTX = aV0Σ −1/2 0 U T 0 USṼ T = WT X̆ = a √ M − 1V0ṼT0 , (A.51)
where X̆ can be regarded as a denoised version of X. The determination of the effective rank K0 ≤ K of the matrix X̆ by using singular values is based on various criteria (Konstantinides & Yao, 1988). Here we choose K0 as follows:
K0 = arg min K′0
 √√√√∑K′0k=1 σ2k∑K
k=1 σ 2 k
≥  , (A.52) where is a positive constant (0 < ≤ 1). Another advantage of a low-rank matrix W is that it can significantly reduce overfitting for learning neural population parameters. In practice, the constraint (A.47) is equivalent to a weight-decay regularization term used in many other optimization problems (Cortes & Vapnik, 1995; Hinton, 2010), which can reduce overfitting to the training data. To prevent the neural networks from overfitting, Srivastava et al. (2014) presented a technique to randomly drop units from the neural network during training, which may in fact be regarded as an attempt to reduce the rank of the weight matrix because the dropout can result in a sparser weights (lower rank matrix). This means that the update is only concerned with keeping the more important components, which is similar to first performing a denoising process by the SVD low rank approximation.
In this stage, we have obtained the optimal parameter W (see A.41). The optimal value of matrix V0 can also be determined, as shown in Appendix A.3.3.
A.3.2 THE 2ND STAGE
For this stage, our goal is to maximize the MI I(Y ;R) and get the optimal parameters θ̃k, k = 1, · · · ,K1. Here the input is y = (y1, · · · , yK1)T and the output r = (r1, · · · , rN )T is also clustered into K1 classes. The responses of Nk neurons in the k-th subpopulation obey a Poisson distribution with mean f̃(eTk y; θ̃k), where ek is a unit vector with 1 in the k-th element and yk = e T k y. By (A.24) and (A.26), we have
〈yk〉yk = 0, (A.53) σ2yk = 〈 y2k 〉 yk = ‖w̃k‖2 . (A.54)
Then for large N , by (1)–(4) and (A.30) we can use the following approximation,
I(Y ;R) ' ĬF = 1
2
〈 ln ( det ( J̆(y)
2πe ))〉 y +H(Y ), (A.55)
where
J̆(y) = diag ( Nα1 |g′1(y1)| 2 , · · · , NαK1 ∣∣g′K1(yK1)∣∣2) , (A.56) g′k(yk) = ∂gk(yk)
∂yk , k = 1, · · · ,K1, (A.57)
gk(yk) = 2 √ f̃(yk; θ̃k), k = 1, · · · ,K1. (A.58)
It is easy to get that
ĬF = 1
2 K1∑ k=1
〈 ln ( Nαk |g′k(yk)| 2
2πe
)〉 y +H(Y )
≤ 1 2 K1∑ k=1
〈 ln ( |g′k(yk)| 2
2πe )〉 y − K1 2 ln ( K1 N ) +H(Y ), (A.59)
where the equality holds if and only if
αk = 1
K1 , k = 1, · · · ,K1, (A.60)
which is consistent with Eq. (A.42).
On the other hand, it follows from the Jensen’s inequality that
ĬF =
〈 ln p (y)−1 det( J̆(y) 2πe )1/2〉 y
≤ ln ∫ det ( J̆(y)
2πe
)1/2 dy, (A.61)
where the equality holds if and only if p (y)−1 det ( J̆(y) )1/2 is a constant, which implies that
p (y) = det ( J̆(y) )1/2 ∫
det ( J̆(y) )1/2 dy = ∏K1 k=1 |g′k(yk)|∫ ∏K1 k=1 |g′k(yk)| dy . (A.62)
From (A.61) and (A.62), maximizing ĨF yields
p (yk) = |g′k(yk)|∫ |g′k(yk)| dyk , k = 1, · · · ,K1. (A.63)
We assume that (A.63) holds, at least approximately. Hence we can let the peak of g′k(yk) be at yk = 〈yk〉yk = 0 and 〈 y2k 〉 yk = σ2yk = ‖w̃k‖ 2. Then combining (A.57), (A.61) and (A.63) we find the optimal parameters θ̃k for the nonlinear functions f̃(yk; θ̃k), k = 1, · · · ,K1.
A.3.3 THE FINAL OBJECTIVE FUNCTION
In the preceding sections we have obtained the initial optimal solutions by maximizing I ( X; Y̆ ) and I(Y ;R). In this section, we will discuss how to find the final optimal V0 and other parameters by maximizing I(X;R) from the initial optimal solutions.
First, we have y = W̃T x̃ = aŷ, (A.64)
where a is given in (A.43) and
ŷ = (ŷ1, · · · , ŷK1)T = CT x̂ = ČT x̌, (A.65) x̂ = Σ −1/2 0 U T 0 x, (A.66) C = VT0 ∈ RK0×K1 , (A.67) x̌ = U0Σ −1/2 0 U T 0 x = U0x̂, (A.68)
Č = U0C = [č1, · · · , čK1 ]. (A.69)
It follows that I(X;R) = I ( X̃;R ) ' ĨG = 1
2
〈 ln ( det ( G(x̂)
2πe ))〉 x̂ +H(X̃), (A.70)
G(x̂) = NŴΦ̂Ŵ T + IK , (A.71)
Ŵ = Σ1/2UTWA1/2 = a √ K−11 I K K0C = √ K−10 I K K0C, (A.72)
where IKK0 is a K ×K0 diagonal matrix with value 1 on the diagonal and
Φ̂ = Φ2, (A.73) Φ = diag (φ(ŷ1), · · · , φ(ŷK1)) , (A.74)
φ(ŷk) = a −1 ∣∣∣∣∂gk(ŷk)∂ŷk ∣∣∣∣ , (A.75) gk(ŷk) = 2 √ f̃(ŷk; θ̃k), (A.76)
ŷk = a −1yk = c T k x̂, k = 1, · · · ,K1. (A.77)
Then we have det (G(x̂)) = det ( NK−10 CΦ̂C T + IK0 ) . (A.78)
For large N and K0/N → 0, we have det (G(x̂)) ≈ det (J(x̂)) = det ( NK−10 CΦ̂C T ) , (A.79)
ĨG ≈ ĨF = −Q− K 2 ln (2πe)− K0 2 ln (ε) +H(X̃), (A.80) Q = −1 2 〈 ln ( det ( CΦ̂C T ))〉 x̂ , (A.81)
ε = K0 N . (A.82)
Hence we can state the optimization problem as:
minimize Q [C] =− 1 2
〈 ln ( det ( CΦ̂C T ))〉
x̂ , (A.83)
subject to CCT = IK0 . (A.84) The gradient from (A.83) is given by:
dQ[C]
dC = −
〈( CΦ̂C T )−1 CΦ̂ + x̂ωT 〉
x̂
, (A.85)
where C = [c1, · · · , cK1 ], ω = (ω1, · · · , ωK1) T , and
ωk = φ(ŷk)φ ′(ŷk)c T k
( CΦ̂C T )−1 ck, k = 1, · · · ,K1. (A.86)
In the following we will discuss how to get the optimal solution of C for two specific cases.
A.4 ALGORITHMS FOR OPTIMIZATION OBJECTIVE FUNCTION
A.4.1 ALGORITHM 1: K0 = K1
Now CCT = CTC = IK1 , then by Eq. (A.83) we have
Q1[C] = − 〈 K1∑ k=1 ln (φ(ŷk)) 〉 x̂ , (A.87)
dQ1[C]
dC = −
〈 x̂ωT 〉 x̂ , (A.88)
ωk = φ′(ŷk)
φ(ŷk) , k = 1, · · · ,K1. (A.89)
Under the orthogonality constraints (Eq. A.84), we can use the following update rule for learning C (Edelman et al., 1998; Amari, 1999):
Ct+1 = Ct + µt dCt
dt , (A.90)
dCt
dt = −dQ1[C
t]
dCt + Ct
( dQ1[C t]
dCt
)T Ct, (A.91)
where the learning rate parameter µt changes with the iteration count t, t = 1, · · · , tmax. Here we can use the empirical average to approximate the integral in (A.88) (see Eq. A.12). We can also apply stochastic gradient descent (SGD) method for online updating of Ct+1 in (A.90).
The orthogonality constraint (Eq. A.84) can accelerate the convergence rate. In practice, the orthogonal constraint (A.84) for objective function (A.83) is not strictly necessary in this case. We can completely discard this constraint condition and consider
minimize Q2 [C] =− 〈 K1∑ k=1 ln (φ (ŷk)) 〉 x̂ − 1 2 ln ( det ( CTC )) , (A.92)
where we assume rank (C) = K1 = K0. If we let
dC dt = −CCT dQ2 [C] dC , (A.93)
then
Tr
( dQ2 [C]
dC
dCT
dt
) = −Tr ( CT dQ2 [C]
dC
dQ2 [C]
dCT C
) ≤ 0. (A.94)
Therefore we can use an update rule similar to Eq. A.90 for learning C. In fact, the method can also be extended to the case K0 > K1 by using the same objective function (A.92).
The learning rate parameter µt (see A.90) is updated adaptively, as follows. First, calculate
µt = vt κt , t = 1, · · · , tmax, (A.95)
κt = 1
K1 K1∑ k=1 ‖∇Ct(:, k)‖ ‖Ct(:, k)‖ , (A.96)
and Ct+1 by (A.90) and (A.91), then calculate the value Q1 [ Ct+1 ] . If Q1 [ Ct+1 ] < Q1[C
t], then let vt+1 ← vt, continue for the next iteration; otherwise, let vt ← τvt, µt ← vt/κt and recalculate Ct+1 and Q1 [ Ct+1 ] . Here 0 < v1 < 1 and 0 < τ < 1 are set as constants. After getting Ct+1 for each update, we employ a Gram–Schmidt orthonormalization process for matrix Ct+1, where the orthonormalization process can accelerate the convergence. However, we can discard the Gram– Schmidt orthonormalization process after iterative t0 (> 1) epochs for more accurate optimization solution C. In this case, the objective function is given by the Eq. (A.92). We can also further optimize parameter b by gradient descent.
When K0 = K1, the objective function Q2 [C] in Eq. (A.92) without constraint is the same as the objective function of infomax ICA (IICA) (Bell & Sejnowski, 1995; 1997), and as a consequence we should get the same optimal solution C. Hence, in this sense, the IICA may be regarded as a special case of our method. Our method has a wider range of applications and can handle more generic situations. Our model is derived by neural populations with a huge number of neurons and it is not restricted to additive noise model. Moreover, our method has a faster convergence rate during training than IICA (see Section 3).
A.4.2 ALGORITHM 2: K0 ≤ K1
In this case, it is computationally expensive to update C by using the gradient of Q (see Eq. A.85), since it needs to compute the inverse matrix for every x̂. Here we provide an alternative method for learning the optimal C. First, we consider the following inequalities.
Proposition 2. The following inequations hold, 1
2
〈 ln ( det ( CΦ̂CT ))〉 x̂ ≤ 1 2 ln ( det ( C 〈 Φ̂ 〉 x̂ CT ))
, (A.97)〈 ln ( det ( CΦCT ))〉 x̂ ≤ ln ( det ( C 〈Φ〉x̂ C T )) (A.98)
≤ 1 2
ln ( det ( C 〈Φ〉2x̂ C T ))
(A.99)
≤ 1 2
ln ( det ( C 〈 Φ̂ 〉 x̂ CT )) , (A.100)
ln ( det ( CΦCT )) ≤ 1 2 ln ( det ( CΦ̂CT )) , (A.101)
where C ∈ RK0×K1 , K0 ≤ K1, and CCT = IK0 .
Proof. Functions ln ( det ( C 〈 Φ̂ 〉 x̂ CT )) and ln ( det ( C 〈Φ〉x̂ CT )) are concave functions about p (x̂) (see the proof of Proposition 5.2. in Huang & Zhang, 2017), which fact establishes inequalities (A.97) and (A.98).
Next we will prove the inequality (A.101). By SVD, we have
CΦ = ÜD̈V̈ T , (A.102)
where Ü is a K0 ×K0 unitary orthogonal matrix, V̈ = [v̈1, v̈2, · · · , v̈K1 ] is an K1 ×K1 unitary orthogonal matrix, and D̈ is an K0×K1 rectangular diagonal matrix with K0 positive real numbers on the diagonal. By the matrix Hadamard’s inequality and Cauchy–Schwarz inequality we have
det ( CΦCTCΦCT ) det ( CΦ̂C T )−1
= det ( D̈V̈ T CTCV̈D̈ T ( D̈D̈ T )−1)
= det ( V̈T1 C TCV̈1 ) = det ( CV̈1
)2 ≤
K0∏ k=1 ( CV̈1 )2 k,k
≤ K0∏ k=1 ( CCT )2 k,k ( V̈T1 V̈1 )2 k,k
= 1, (A.103)
where V̈1 = [v̈1, v̈2, · · · , v̈K0 ] ∈ RK1×K0 . The last equality holds because of CC T = IK0 and V̈T1 V̈1 = IK0 . This establishes inequality (A.101) and the equality holds if and only if K0 = K1 or CV̈1 = IK0 .
Similarly, we get inequality (A.99):
ln ( det ( C 〈Φ〉x̂ C T )) ≤ 1 2 ln ( det ( C 〈Φ〉2x̂ C T )) . (A.104)
By Jensen’s inequality, we have 〈φ (ŷk)〉2x̂ ≤ 〈 φ (ŷk) 2 〉
x̂ , ∀k = 1, · · · ,K1. (A.105)
Then it follows from (A.105) that inequality (A.100) holds:
1 2 ln ( det ( C 〈Φ〉2x̂ C T )) ≤ 1 2 ln ( det ( C 〈 Φ̂ 〉 x̂ CT )) . (A.106)
This completes the proof of Proposition 2.
By Proposition 2, if K0 = K1 then we get 1
2
〈 ln ( det ( Φ̂ ))〉
x̂ ≤ 1 2 ln ( det (〈 Φ̂ 〉 x̂ )) , (A.107)
〈ln (det (Φ))〉x̂ ≤ ln (det (〈Φ〉x̂)) (A.108)
= 1 2 ln ( det ( 〈Φ〉2x̂ )) (A.109) ≤ 1 2 ln ( det (〈 Φ̂ 〉 x̂ )) , (A.110)
ln (det (Φ)) = 1 2 ln ( det ( Φ̂ )) . (A.111)
On the other hand, it follows from (A.81) and Proposition 2 that〈 ln ( det ( CΦCT ))〉 x̂ ≤ −Q ≤ 1 2 ln ( det ( C 〈 Φ̂ 〉 x̂ CT ))
, (A.112)〈 ln ( det ( CΦCT ))〉 x̂ ≤ −Q̂ ≤ 1 2 ln ( det ( C 〈 Φ̂ 〉 x̂ CT )) . (A.113)
Hence we can see that Q̂ is close to Q (see A.81). Moreover, it follows from the Cauchy–Schwarz inequality that 〈
(Φ)k,k 〉 x̂ = 〈φ (ŷk)〉ŷk ≤ (∫ φ (ŷk) 2 dŷk ∫ p (ŷk) 2 dŷk )1/2 , (A.114)
where k = 1, · · · ,K1, the equality holds if and only if the following holds:
p (ŷk) = φ (ŷk)∫ φ (ŷk) dŷk , k = 1, · · · ,K1, (A.115)
which is the similar to Eq. (A.63).
Since I(X;R) = I(Y ;R) (see Proposition 1), by maximizing I(X;R) we hope the equality in inequality (A.61) and equality (A.63) hold, at least approximatively. On the other hand, let
Copt = arg min C Q[C] = arg max C
(〈 ln ( det(CΦ̂CT ) )〉
x̂
) , (A.116)
Ĉopt = arg min C Q̂[C] = arg max C
( ln ( det ( C 〈Φ〉2x̂ C T ))) , (A.117)
Copt and Ĉopt make (A.63) and (A.115) to hold true, which implies that they are the same optimal solution: Copt = Ĉopt.
Therefore, we can use the following objective function Q̂[C] as a substitute for Q[C] and write the optimization problem as:
minimize Q̂[C] =− 1 2
ln ( det ( C 〈Φ〉2x̂ C T )) , (A.118)
subject to CCT = IK0 . (A.119) The update rule (A.90) may also apply here and a modified algorithm similar to Algorithm 1 may be used for parameter learning.
A.5 SUPPLEMENTARY EXPERIMENTS
A.5.1 QUANTITATIVE METHODS FOR COMPARISON
To quantify the efficiency of learning representations by the above algorithms, we calculate the coefficient entropy (CFE) for estimating coding cost as follows (Lewicki & Olshausen, 1999; Lewicki & Sejnowski, 2000):
y̌k = ζw̌ T k x̌, k = 1, · · · ,K1, (A.120)
ζ = K1∑K1
k=1 ‖w̌k‖ , (A.121)
where x̌ is defined by Eq. (A.68), and w̌k is the corresponding optimal filter. To estimate the probability density of coefficients qk(y̌k) (k = 1, · · · ,K1) from the M training samples, we apply the kernel density estimation for qk(y̌k) and use a normal kernel with an adaptive optimal window width. Then we define the CFE h as
h = 1
K1 K1∑ k=1 Hk(Y̌k), (A.122)
Hk(Y̌k) = −∆ ∑ nqk(n∆) log2 qk(n∆), (A.123)
where qk(y̌k) is quantized as discrete qk(n∆) and ∆ is the step size.
Methods such as IICA and SRBM as well as our methods have feedforward structures in which information is transferred directly through a nonlinear function, e.g., the sigmoid function. We can use the amount of transmitted information to measure the results learned by these methods. Consider a neural population with N neurons, which is a stochastic system with nonlinear transfer functions. We chose a sigmoidal transfer function and Gaussian noise with standard deviation set to 1 as the system noise. In this case, from (1), (A.8) and (A.11), we see that the approximate MI IG is equivalent to the case of the Poisson neuron model. It follows from (A.70)–(A.82) that
I(X;R) = I ( X̃;R ) = H(X̃)−H ( X̃|R ) ' ĨG = H(X̃)− h1, (A.124)
H ( X̃|R ) ' h1 = − 1
2
〈 ln ( det ( 1
2πe
( NK−10 CΦ̂C T + IK0 )))〉 x̂ , (A.125)
where we set N = 106. A good representation should make the MI I(X;R) as big as possible. Equivalently, for the same inputs, a good representation should make the conditional entropy (CDE) H ( X̃|R ) (or h1) as small as possible.
A.5.2 COMPARISON OF BASIS VECTORS
We compared our algorithm with an up-to-date sparse coding algorithm, the mini-batch dictionary learning (MBDL) as given in (Mairal et al., 2009; 2010) and integrated in Python library, i.e. scikitlearn. The input data was the same as the above, i.e. 105 nature image patches preprocessed by the ZCA whitening filters.
We denotes the optimal dictionary learned by MBDL as B̌ ∈ RK×K1 for which each column represents a basis vector. Now we have
x ≈ UΣ1/2UT B̌y = B̃y, (A.126) B̃ = UΣ1/2UT B̌, (A.127)
where y = (y1, · · · , yK1) T is the coefficient vector.
Similarly, we can obtain a dictionary from the filter matrix C. Suppose rank (C) = K0 ≤ K1, then it follows from (A.64) that
x̂ = ( aCCT )−1 Cy. (A.128)
By (A.66) and (A.128), we get
x ≈ By = aBCTΣ−1/20 UT0 x, (A.129)
B = a−1U0Σ 1/2 0
( CCT )−1 C = [b1, · · · ,bK1 ] , (A.130)
where y = WTx = aCTΣ−1/20 U T 0 x, the vectors b1, · · · ,bK1 can be regarded as the basis vectors and the strict equality holds when K0 = K1 = K. Recall that X = [x1, · · · , xM ] = USṼ T (see Eq. A.49) and Y = [y1, · · · , yM ] = WTX = a √ M − 1CT ṼT0 , then we get X̆ = BY =√
M − 1U0Σ1/20 ṼT0 ≈ X. Hence, Eq. (A.129) holds. The basis vectors shown in Figure 4(a)–4(e) correspond to filters in Figure 2(a)–2(e). And Figure 4(f) illustrates the optimal dictionary B̃ learned by MBDL, where we set the regularization parameter as λ = 1.2/ √ K, the batch size as 50 and the total number of iterations to perform as 20000, which took about 3 hours for training. From Figure 4 we see that these basis vectors obtained by the above algorithms have local Gabor-like shapes except for those by SRBM. If rank(B̌) = K = K1, then the matrix B̌−T can be regarded as a filter matrix like matrix Č (see Eq. A.69). However, from the column vector of matrix B̌−T we cannot find any local Gabor-like filter that resembles the filters shown in Figure 2. Our algorithm has less computational cost and a much faster convergence rate than the sparse coding algorithm. Moreover, the sparse coding method involves a dynamic generative model that requires relaxation and is therefore unsuitable for fast inference, whereas the feedforward framework of our model is easy for inference because it only requires evaluating the nonlinear tuning functions.
A.5.3 LEARNING OVERCOMPLETE BASES
We have trained our model on the Olshausen’s nature image patches with a highly overcomplete setup by optimizing the objective (A.118) by Alg.2 and got Gabor-like filters. The results of 400 typical filters chosen from 1024 output filters are displayed in Figure 5(a) and corresponding base (see Eq. A.130) are shown in Figure 5(b). Here the parameters are K1 = 1024, tmax = 100, v1 = 0.4, τ = 0.8, and = 0.98 (see A.52), from which we got rank (B) = K0 = 82. Compared to the ICA-like results in Figure 2(a)–2(c), the average size of Gabor-like filters in Figure 5(a) is bigger, indicating that the small noise-like local structures in the images have been filtered out.
We have also trained our model on 60,000 images of handwritten digits from MNIST dataset (LeCun et al., 1998) and the resultant 400 typical optimal filters and bases are shown in Figure 5(c) and Figure 5(d), respectively. All parameters were the same as Figure 5(a) and Figure 5(b): K1 = 1024, tmax = 100, v1 = 0.4, τ = 0.8 and = 0.98, from which we got rank (B) = K0 = 183. From these figures we can see that the salient features of the input images are reflected in these filters and bases. We could also get the similar overcomplete filters and bases by SRBM and MBDL. However, the results depended sensitively on the choice of parameters and the training took a long time.
Figure 6 shows that CFE as a function of training time for Alg.2, where Figure 6(a) corresponds to Figure 5(a)-5(b) for learning nature image patches and Figure 6(b) corresponds to Figure 5(c)-5(d) for learning MNIST dataset. We set parameters tmax = 100 and τ = 0.8 for all experiments and varied parameter v1 for each experiment, with v1 = 0.2, 0.4, 0.6 or 0.8. These results indicate a fast convergence rate for training on different datasets. Generally, the convergence is insensitive to the change of parameter v1.
We have also performed additional tests on other image datasets and got similar results, confirming the speed and robustness of our learning method. Compared with other methods, e.g., IICA, FICA, MBDL, SRBM or sparse autoencoders etc., our method appeared to be more efficient and robust for unsupervised learning of representations. We also found that complete and overovercomplete filters and bases learned by our methods had local Gabor-like shapes while the results by SRBM or MBDL did not have this property.
A.5.4 IMAGE DENOISING
Similar to the sparse coding method applied to image denoising (Elad & Aharon, 2006), our method (see Eq. A.130) can also be applied to image denoising, as shown by an example in Figure 7. The filters or bases were learned by using 7×7 image patches sampled from the left half of the image, and subsequently used to reconstruct the right half of the image which was distorted by Gaussian noise. A common practice for evaluating the results of image denoising is by looking at the difference between the reconstruction and the original image. If the reconstruction is perfect the difference should look like Gaussian noise. In Figure 7(c) and 7(d) a dictionary (100 bases) was learned by MBDL and orthogonal matching pursuit was used to estimate the sparse solution. 1 For our method (shown in Figure 7(b)), we first get the optimal filters parameter W, a low rank matrix (K0 < K), then from the distorted image patches xm (m = 1, · · · ,M ) we get filter outputs ym = WTxm and the reconstruction x̆m = Bym (parameters: = 0.975 and K0 = K1 = 14). As can be seen from Figure 7, our method worked better than dictionary learning, although we only used 14 bases compared with 100 bases used by dictionary learning. Our method is also more efficient. We can get better optimal bases B by a generative model using our infomax approach (details not shown).
1Python source code is available at http://scikit-learn.org/stable/ downloads/plot image denoising.py

This paper presents an information theoretic framework for unsupervised learning. The framework relies on infomax principle, whose goal is to maximize the mutual information between input and output. The authors propose a two-step algorithm for learning in this setting. First, by leveraging an asymptotic approximation to the mutual information, the global objective is decoupled into two subgoals whose solutions can be expressed in closed form. Next, these serve as the initial guess for the global solution, and are refined by the gradient descent algorithm.
While the story of the paper and the derivations seem sound, the clarity and presentation of the material could improve. For example, instead of listing step by step derivation of each equation, it would be nice to first give a high-level presentation of the result and maybe explain briefly the derivation strategy. The very detailed aspects of derivations, which could obscure the underlying message of the result could perhaps be postponed to later sections or even moved to an appendix.
A few questions that the authors may want to clarify:
1. Page 4, last paragraph: "from above we know that maximizing I(X;R) will result in maximizing I(Y;R) and I(X,Y^U)". While I see the former holds due to equality in 2.20, the latter is related via a bound in 2.21. Due to the possible gap between I(X;R) and I(X,Y^U), can your claim that maximizing of the former indeed maximizes the latter be true?
2. Paragraph above section 2.2.2: it is stated that, dropout used to prevent overfitting may in fact be regarded as an attempt to reduce the rank of the weight matrix. No further tip is provided why this should be the case. Could you elaborate on that?
3. At the end of page 9: "we will discuss how to get optimal solution of C for two specific cases". If I understand correctly, you actually are not guaranteed to get the optimal solution of C in either case, and the best you can guarantee is reaching a local optimum. This is due to the nonconvexity of the constraint 2.80 (quadratic equality). If optimality cannot be guaranteed, please correct the wording accordingly.


EMERGENCE OF FOVEAL IMAGE SAMPLING FROM LEARNING TO ATTEND IN VISUAL SCENES
We describe a neural attention model with a learnable retinal sampling lattice. The model is trained on a visual search task requiring the classification of an object embedded in a visual scene amidst background distractors using the smallest number of fixations. We explore the tiling properties that emerge in the model’s retinal sampling lattice after training. Specifically, we show that this lattice resembles the eccentricity dependent sampling lattice of the primate retina, with a high resolution region in the fovea surrounded by a low resolution periphery. Furthermore, we find conditions where these emergent properties are amplified or eliminated providing clues to their function.
1 INTRODUCTION
A striking design feature of the primate retina is the manner in which images are spatially sampled by retinal ganglion cells. Sample spacing and receptive fields are smallest in the fovea and then increase linearly with eccentricity, as shown in Figure 1. Thus, we have highest spatial resolution at the center of fixation and lowest resolution in the periphery, with a gradual fall-off in resolution as one proceeds from the center to periphery. The question we attempt to address here is, why is the retina designed in this manner - i.e., how is it beneficial to vision?
The commonly accepted explanation for this eccentricity dependent sampling is that it provides us with both high resolution and broad coverage of the visual field with a limited amount of neural resources. The human retina contains 1.5 million ganglion cells, whose axons form the sole output of the retina. These essentially constitute about 300,000 distinct samples of the image due to the multiplicity of cell types coding different aspects such as on vs. off channels (Van Essen & Anderson, 1995). If these were packed uniformly at highest resolution (120 samples/deg, the Nyquist-dictated sampling rate corresponding to the spatial-frequencies admitted by the lens), they would subtend an image area spanning just 5x5 deg2. Thus we would have high-resolution but essentially tunnel vision. Alternatively if they were spread out uniformly over the entire monocular visual field spanning roughly 150 deg2 we would have wide field of coverage but with very blurry vision, with each sample subtending 0.25 deg (which would make even the largest letters on a Snellen eye chart illegible). Thus, the primate solution makes intuitive sense as a way to achieve the best of both of these worlds. However we are still lacking a quantitative demonstration that such a sampling strategy emerges as the optimal design for subserving some set of visual tasks.
Here, we explore what is the optimal retinal sampling lattice for an (overt) attentional system performing a simple visual search task requiring the classification of an object. We propose a learnable retinal sampling lattice to explore what properties are best suited for this task. While evolutionary pressure has tuned the retinal configurations found in the primate retina, we instead utilize gradient descent optimization for our in-silico model by constructing a fully differentiable dynamically controlled model of attention.
Our choice of visual search task follows a paradigm widely used in the study of overt attention in humans and other primates (Geisler & Cormack, 2011). In many forms of this task, a single target is randomly located on a display among distractor objects. The goal of the subject is to find the target as rapidly as possible. Itti & Koch (2000) propose a selection mechanism based on manually
Published as a conference paper at ICLR 2017
Ill? V. A. Perry PI al. cells is shown in Fig. Q(B). In this case the slopes of the regression lines of cell body size on the logarithm of cell density for the nasal (a = 43.4, h = -6.87) and the temporal cells (u = 53.6, h = 9.1) were not significantly different (r = 1.0, df‘ 54). Thus the differences in cell body size of PX cells in different parts of the retina is clearly related to the variation in ganglion cell density, although the differences in dendritic field size is not simply related to cell density alone. The correlation coefficient for the dendritic field size of both nasal and temporal Pa cells and the logarithm of the ganglion cell density was 0.97, i.e. most of the variance in dendritic field size can be attributed to differences in ganglion cell density. As a check on the accuracy of our ganglion cell counts we used them to estimate the total number of ganglion cells in the macaque retina which can, in turn, be compared with published estimation of the number of axons in the optic nerve. To do this we plotted onto a scaled drawing of one Nissl-stained whole-mounted retina (670mm’ in area) the isodensity contours estimated from the counts along the A meridia. The isodensity contours were drawn as approximately ellipsoid but tapering in the nasal retina as has been shown in previous studies of peripheral ganglion cell counts of the primate retina.6’,“x The area between adjacent contours was multiplied by the mean ceil density of the two isodensity contours. This was repeated for the area between all pairs of contours and the totals for all areas were summed to give the total number of ganglion cells. This yielded an estimate of 1.4 x 10” ganglion cells in the retina, in good agreement with estimates of the number of axons in the optic nerve i.e. 1.5-1.8 x IOh, I.4 x I O' and 1.2-1.3 x 10”.40~so.” If we neglect the naso-temporal overlap, which is small in primates,h.h’ then approximately 60’1/ of the cells lie in the nasal retina and 40“;) in the temporal retina. Our intention was to show which cell types project to the magno- and parvocellufar layers of the lateraf geniculate nucleus and to estimate the percentage of retinal ganglion cells at different eccentricities which
0 1 2 3 L 5 6 7 8 9 10 li 12 13 I‘ ECCENTRICITY fTH?
defined low level features of real images to locate various search targets. Here the neural network must learn what features are most informative for directing attention.
While neural attention models have been applied successfully to a variety of engineering applications (Bahdanau et al., 2014; Jaderberg et al., 2015; Xu et al., 2015; Graves et al., 2014), there has been little work in relating the properties of these attention mechanisms back to biological vision. An important property which distinguishes neural networks from most other neurobiological models is their ability to learn internal (latent) features directly from data.
But existing neural network models specify the input sampling lattice a priori. Larochelle & Hinton (2010) employ an eccentricity dependent sampling lattice mimicking the primate retina, and Mnih et al. (2014) utilize a multi scale glimpse window’ that forms a piece-wise approximation of this scheme. While it seems reasonable to think that these design choices contribute to the good performance of these systems, it remains to be seen if this arrangement emerges as the optimal solution.
We further extend the learning paradigm of neural networks to the structural features of the glimpse mechanism of an attention model. To explore emergent properties of our learned retinal configurations, we train on artificial datasets where the factors of variation are easily controllable. Despite this departure from biology and natural stimuli, we find our model learns to create an eccentricity dependent layout where a distinct central region of high acuity emerges surrounded by a low acuity periphery. We show that the properties of this layout are highly dependent on the variations present in the task constraints. When we depart from physiology by augmenting our attention model with the ability to spatially rescale or zoom on its input, we find our model learns a more uniform layout which has properties more similar to the glimpse window proposed in Jaderberg et al. (2015); Gregor et al. (2015). These findings help us to understand the task conditions and constraints in which an eccentricity dependent sampling lattice emerges.
2 RETINAL TILING IN NEURAL NETWORKS WITH ATTENTION
Attention in neural networks may be formulated in terms of a differentiable feedforward function. This allows the parameters of these models to be trained jointly with backpropagation. Most formulations of visual attention over the input image assume some structure in the kernel filters. For example, the recent attention models proposed by Jaderberg et al. (2015); Mnih et al. (2014); Gregor et al. (2015); Ba et al. (2014) assume each kernel filter lies on a rectangular grid. To create a learnable retinal sampling lattice, we relax this assumption by allowing the kernels to tile the image independently.
2.1 GENERATING A GLIMPSE
We interpret a glimpse as a form of routing where a subset of the visual scene U is sampled to form a smaller output glimpse G. The routing is defined by a set of kernels k[•](s), where each kernel i specifies which part of the input U [•] will contribute to a particular output G[i]. A control variable s
is used to control the routing by adjusting the position and scale of the entire array of kernels. With this in mind, many attention models can be reformulated into a generic equation written as
G[i] = H∑ n W∑ m U [n,m]k[m,n, i](s) (1)
where m and n index input pixels of U and i indexes output glimpse features. The pixels in the input image U are thus mapped to a smaller glimpse G.
2.2 RETINAL GLIMPSE
The centers of each kernel filter µ́[i] are calculated with respect to control variables sc and sz and learnable offset µ[i]. The control variables specify the position and zoom of the entire glimpse. µ[i] and σ[i] specify the position and spread respectively of an individual kernel k[−,−, i]. These parameters are learned during training with backpropagation. We describe how the control variables are computed in the next section. The kernels are thus specified as follows:
µ́[i] = (sc − µ[i])sz (2) σ́[i] = σ[i]sz (3)
k[m,n, i](s) = N (m; µ́x[i], σ́[i])N (n; µ́y[i], σ́[i]) (4)
We assume kernel filters factorize between the horizontal m and vertical n dimensions of the input image. This factorization is shown in equation 4, where the kernel is defined as an isotropic gaussian N . For each kernel filter, given a center µ́[i] and scalar variance σ́[i], a two dimensional gaussian is defined over the input image as shown in Figure 2. These gaussian kernel filters can be thought of as a simplified approximation to the receptive fields of retinal ganglion cells in primates (Van Essen & Anderson, 1995).
While this factored formulation reduces the space of possible transformations from input to output, it can still form many different mappings from an input U to outputG. Figure 3B shows the possible windows which an input image can be mapped to an output G. The yellow circles denote the central location of a particular kernel while the size denotes the standard deviation. Each kernel maps to one of the outputs G[i].
Positional control sc can be considered analogous to the motor control signals which executes saccades of the eye, whereas sz would correspond to controlling a zoom lens in the eye (which has no counterpart in biology). In contrast, training defines structural adjustments to individual kernels which include its position in the lattice as well as its variance. These adjustments are only possible during training and are fixed afterwards.Training adjustments can be considered analagous to the incremental adjustments in the layout of the retinal sampling lattice which occur over many generations, directed by evolutionary pressure in biology.
Recurrent ht
Glimpse Gt
Control sc,t ; sz,t
B. Controlling the Retinal Lattice
Final LayoutInitial Layout Training A. Optimizing the Retinal Lattice Time t t+1 t+2 Recurrent ht+1 Glimpse Gt+1 Control sc,t+1 ; sz,t+1 Recurrent ht+2 Glimpse Gt+2 Control sc,t+2 ; sz,t+2
Figure 3: A: Starting from an initial lattice configuration of a uniform grid of kernels, we learn an optmized configuration from data. B: Attentional fixations generated during inference in the model, shown unrolled in time (after training).
3 RECURRENT NEURAL ARCHITECTURE FOR ATTENTION
A glimpse at a specific timepoint, Gt, is processed by a fully-connected recurrent network frnn().
ht = frnn(Gt, ht−1) (5) [sc,t; sz,t] = fcontrol(ht) (6)
The global center sc,t and zoom sz,t are predicted by the control network fcontrol() which is parameterized by a fully-connected neural network.
In this work, we investigate three variants of the proposed recurrent model:
• Fixed Lattice: The kernel parameters µ[i] and σ[i] for each retinal cell are not learnable. The model can only translate the kernel filters sc,t = fcontrol(ht) and the global zoom is fixed sz,t = 1. • Translation Only: Unlike the fixed lattice model, µ[i] and σ[i] are learnable (via back-
propagation). • Translation and Zoom: This model follows equation 6 where it can both zoom and trans-
late the kernels.
A summary for comparing these variants is shown in Table 1.
Prior to training, the kernel filters are initialized as a 12x12 grid (144 kernel filters), tiling uniformly over the central region of the input image and creating a retinal sampling lattice as shown in Figure 5 before training. Our recurrent network, frnn is a two layer traditional recurrent network with 512- 512 units. Our control network, fcontrol is a fully-connected network with 512-3 units (x,y,zoom)
in each layer. Similarly, our prediction networks are fully-connected networks with 512-10 units for predicting the class. We use ReLU non-linearities for all hidden unit layers.
Our model as shown in Figure 3C are differentiable and trained end-to-end via backpropagation through time. Note that this allows us to train the control network indirectly from signals backpropagated from the task cost. For stochastic gradient descent optimization we use Adam (Kingma & Ba, 2014) and construct our models in Theano (Bastien et al., 2012).
4 DATASETS AND TASKS
4.1 MODIFIED CLUTTERED MNIST DATASET
Example images from of our dataset are shown in Figure 4. Handwritten digits from the original MNIST dataset LeCun & Cortes (1998) are randomly placed over a 100x100 image with varying amounts of distractors (clutter). Distractors are generated by extracting random segments of nontarget MNIST digits which are placed randomly with uniform probability over the image. In contrast to the cluttered MNIST dataset proposed in Mnih et al. (2014), the number of distractors for each image varies randomly from 0 to 20 pieces. This prevents the attention model from learning a solution which depends on the number ‘on’ pixels in a given region. In addition, we create another dataset (Dataset 2) with an additional factor of variation: the original MNIST digit is randomly resized by a factor of 0.33x to 3.0x. Examples of this dataset are shown in the second row of Figure 4.
4.2 VISUAL SEARCH TASK
We define our visual search task as a recognition task in a cluttered scene. The recurrent attention model we propose must output the class ĉ of the single MNIST digit appearing in the image via the prediction network fpredict(). The task loss, L, is specified in equation 8. To minimize the classification error, we use cross-entropy cost:
ĉt,n = fpredict(ht,n) (7)
L = N∑ n T∑ t cnlog(ĉt,n) (8)
Analolgous to the visual search experiments performed in physiological studies, we pressure our attention model to accomplish the visual search as quickly as possible. By applying the task loss to every timepoint, the model is forced to accurately recognize and localize the target MNIST digit in as few iterations as possible. In our classification experiments, the model is given T = 4 glimpses.
Before Training (Initial Layout) After 1 epochs After 10 epochs After 100 epochs
5 RESULTS
Figure 5shows the layouts of the learned kernels for a Translation Only model at different stages during training. The filters are smoothly transforming from a uniform grid of kernels to an eccentricity dependent lattice. Furthermore, the kernel filters spread their individual centers to create a sampling lattice which covers the full image. This is sensible as the target MNIST digit can appear anywhere in the image with uniform probability.
When we include variable sized digits as an additional factor in the dataset, the translation only model shows an even greater diversity of variances for the kernel filters. This is shown visually in the first row of Figure 6. Furthermore, the second row shows a highly dependent relationship between the sampling interval and standard deviatoin of the retinal sampling lattice and eccentricity from the center. This dependency increases when training on variable sized MNIST digits (Dataset 2). This
relationship has also been observed in the primate visual system (Perry et al., 1984; Van Essen & Anderson, 1995).
When the proposed attention model is able to zoom its retinal sampling lattice, a very different layout emerges. There is much less diversity in the distribution of kernel filter variances as evidenced in Figure 6. Both the sampling interval and standard deviation of the retinal sampling lattice have far less of a dependence on eccentricity. As shown in the last column of Figure 6, we also trained this model on variable sized digits and noticed no significant differences in sampling lattice configuration.
Figure 7 shows how each model variant makes use of its retinal sampling lattice after training. The strategy each variant adopts to solve the visual search task helps explain the drastic difference in lattice configuration. The translation only variant simply translates its high acuity region to recognize and localize the target digit. The translation and zoom model both rescales and translates its sampling lattice to fit the target digit. Remarkably, Figure 7 shows that both models detect the digit early on and make minor corrective adjustments in the following iterations.
Table 2 compares the classification performance of each model variant on the cluttered MNIST dataset with fixed sized digits (Dataset 1). There is a significant drop in performance when the retinal sampling lattice is fixed and not learnable, confirming that the model is benefitting from learning the high-acuity region. The classification performance between the Translation Only and Translation and Zoom model is competitive. This supports the hypothesis that the functionality of a high acuity region with a low resolution periphery is similar to that of zoom.
6 CONCLUSION
When constrained to a glimpse window that can translate only, similar to the eye, the kernels converge to a sampling lattice similar to that found in the primate retina (Curcio & Allen, 1990; Van Essen & Anderson, 1995). This layout is composed of a high acuity region at the center surrounded by a wider region of low acuity. Van Essen & Anderson (1995) postulate that the linear relationship between eccentricity and sampling interval leads to a form of scale invariance in the primate retina. Our results from the Translation Only model with variable sized digits supports this conclusion. Additionally, we observe that zoom appears to supplant the need to learn a high acuity region for the visual search task. This implies that the high acuity region serves a purpose resembling that of a zoomable sampling lattice. The low acuity periphery is used to detect the search target and the high acuity ‘fovea’ more finely recognizes and localizes the target. These results, while obtained on an admittedly simplified domain of visual scenes, point to the possibility of using deep learning as a tool to explore the optimal sample tiling for a retinal in a data driven and task-dependent manner. Exploring how or if these results change for more challenging tasks in naturalistic visual scenes is a future goal of our research.
ACKNOWLEDGMENTS
We would like to acknowledge everyone at the Redwood Center for their helpful discussion and comments. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Tesla K40 GPUs used for this research.

This paper presents a succinct argument that the principle of optimizing receptive field location and size in a simulated eye that can make saccades with respect to a classification error of images of data whose labels depend on variable-size and variable-location subimages, explains the existence of a foveal area in e.g. the primate retina.
The argument could be improved by using more-realistic image data and drawing more direct correspondence with the number, receptive field sizes and eccentricities of retinal cells in e.g. the macaque, but the authors would then face the challenge of identifying a loss function that is both biologically plausible and supportive of their claim.
The argument could also be improved by commenting on the timescales involved. Presumably the density of the foveal center depends on the number of of saccades allowed by the inference process, as well as the size of the target sub-images, and also has an impact on the overall classification accuracy.
Why does the classification error rate of dataset 2 remain stubbornly at 24%? This seems so high that the model may not be working the way we’d like it to. It seems that the overall argument of the paper pre-supposes that the model can be trained to be a good classifier. If there are other training strategies or other models that work better and differently, then it raises the question of why do our eyes and visual cortex not work more like *those ones* if evolutionary pressures are applying the same pressure as our training objective.
Why does the model with zooming powers out-do the translation-only model on dataset 1 (where all target images are the same size) and tie the translation-only model dataset 2 (where the target images have different sizes, for which the zooming model should be tailor-made?). Between this strange tie and the high classification rate on Dataset 2, I wonder if maybe one or both models isn’t being trained to its potential, which would undermine the overall claim.
Comparing this model to other attention models (e.g. spatial transformer networks, DRAW) would be irrelevant to what I take to be the main point of the paper, but it would address the potential concerns above that training just didn’t go very well, or there was some problem with the model parameterization that could be easily fixed.