In recent years, many deep architectures have been proposed in different fields. However, to obtain good results, most of the previous deep models need a large number of training data. In this paper, for small and middle scale applications, we propose a novel deep learning framework based on stacked feature learning models. Particularly, we stack marginal Fisher analysis (MFA) layer by layer for the initialization of the deep architecture and call it “Marginal Deep Architectures” (MDA). In the implementation of MDA, the weight matrices of MFA are first learned layer by layer, and then we exploit some deep learning techniques, such as back propagation, dropout and denoising to fine tune the network. To evaluate the effectiveness of MDA, we have compared it with some feature learning methods and deep learning models on 7 small and middle scale real-world applications, including handwritten digits recognition, speech recognition, historical document understanding, image classification, action recognition and so on. Extensive experiments demonstrate that MDA performs not only better than shallow feature learning models, but also state-of-the-art deep learning models in these applications.
1 INTRODUCTION
Deep learning methods have achieved desirable performance in many domains, such as image classification and detection, document analysis and recognition, natural language processing, video analysis (Krizhevsky et al., 2012; Chan et al., 2014; Ciresan et al., 2010; Collobert & Weston, 2008; Le et al., 2011). Deep learning methods learn the data representation by using multiple processing layers, which discover the intricate structure of high dimensional data with multiple levels of abstraction (LeCun et al., 2015). For example, for face recognition, the learned features of first layer may be the edges, directions and some local information. The second layer typically detects some object parts which are combination of edges and directions. The higher layers may further abstract the face image by combining the features of previous layers (outline of the eyes, nose, lips). This procedure is very similar with human visual and perceptual system.
In recently years, many deep learning methods have been proposed (l. Boureau & others, 2008; Lee et al., 2009b;a; Hinton & Salakhutdinov, 2006). However, most models meet some difficult problems to solve, such as some parameters need to be randomly initialized, like the weight matrix of two successive layers in deep belief networks (DBNs) and the convolution kernel in convolutional neural networks (CNNs). In addition, traditional deep learning methods need a large scale training data to train the complex networks. It causes many problems in the training process. If we don’t initialize the parameters properly, the optimization procedure might need a long training time and fall into local minima. Alternatively, many feature learning models have been proposed to learn the intrinsic structures of high-dimensional data and avoid the curse of dimensionality. In particular, most of them can be trained with small and middle scale of data and their learning algorithms are generally based on closed-form solution or convex optimization. For instance, marginal Fisher analysis (MFA) (Yan et al., 2007; Zhong et al., 2013) is one of the feature learning models that is a supervised method based on the graph embedding framework. It utilizes an intrinsic graph to characterize the intraclass compactness, and another penalty graph to characterize the interclass separability. Its optimal solution can be learned by generalized eigenvalue decomposition. However,
on the one hand, shallow feature learning models cannot work well on the data with highly nonlinear structure; on the other hand, few efforts are made to combine shallow feature learning models for the design of deep architectures.
In order to simultaneously solve the existing problems in deep learning methods and combine the advantages of feature learning models, we proposed a novel deep learning method based on stacked feature learning models. Particularly, we stack marginal Fisher analysis (MFA) layer by layer for the initialization of the deep architecture and call it “Marginal Deep Architectures” (MDA). Firstly, the input data are mapped to higher dimensional space by using random weight matrix. Then we use MFA to learn the lower dimensional representation layer by layer. In the implementation of this architecture, we add some tricks in the training process, such as back propagation, dropout and denoising to fine tune the network. Finally, the softmax layer is connected to the last feature layer. We have compared our MDA with some feature learning methods and deep learning models on different domains of datasets (including handwritten digits recognition, speech recognition, historical document understanding, image classification, action recognition and so on). Extensive experiments demonstrate that MDA performs not only better than shallow feature learning models, but also state-of-the-art deep learning models in small and middle scale applications.
The contributions of this work are highlighted as follows.
1. We propose a novel structure to build a deep architecture. The first hidden layer has twice or quadruple neurons as the input layer. Then we can use some feature learning models layer by layer to learn the compact representations of data. Finally, we set the last layer as a softmax classifier.
2. Traditional deep learning models in general need a large scale training data. Compared with traditional deep learning models, MDA can work better than traditional deep learning models in small and middle scale applications because the initialization of the weight matrices using MFA is much better than that using random initialization.
3. Our MDA can work well in different domains of datasets, such as handwritten digits, spoken letters and natural images. Extensive experiments demonstrate that MDA is a general model to handel small and middle scale data. On the other hand, for large scale datasets, like CIFAR-10, MDA works comparatively with other deep learning methods.
The rest of this paper is organized as follows: In Section 2, we give a brief overview of related work. In Section 3, we present the marginal Fisher analysis (MFA) and the proposed marginal deep architectures (MDA) in detail. The experimental settings and results are reported in Section 4, while Section 5 concludes this paper with remarks and future work.
2 RELATED WORK
With the development of deep learning methods, many deep networks have been proposed in recent years (Donahue et al., 2013; Krizhevsky et al., 2012; Long et al., 2015; Zhou et al., 2014). These deep learning models show their powerful performance in various fields, such as image classification and analysis, document analysis and recognition, natural language processing et al. In the area of image analysis, Hinton et al. proposed a large, deep convolutional neural network (Alex net) to classify the 1.2 million high-resolution images in the ImageNet. It uses efficient GPU to speed their method. The results show that a large, deep convolutional neural network is capable of achieving recordbreaking results on a highly challenging dataset using purely supervised learning (Krizhevsky et al., 2012). In order to popularize the deep convolutional neural network, Donahue ea al. proposed DeCAF (Deep Convolutional Activation Feature) which is trained in a fully supervised fashion on a large, fixed set of object recognition tasks (Donahue et al., 2013). DeCAF provides a uniform framework for researchers who can improve and change this framework on some specific tasks. However, its performance at scene recognition has not attained the same level of success. In order to handle this problem, Zhou et al. introduce a new scene-centric database called Places with over 7 million labeled pictures of scenes. Then, they learn the deep features for scene recognition tasks by using the same architecture as ImageNet, and establish new state-of-the-art results on several scenecentric datasets (Zhou et al., 2014). However, these methods based on convolutional operation need very large scale training samples and a long training time. They can not work well on small and middle scale applications.
In other domains, deep learning methods also achieve good performance. Hinton et al. represent the shared views of four research groups that have had recent successes in using DNNs for automatic speech recognition (ASR). The DNNs that contain many layers of nonlinear hidden units and a very large output layer can outperform Gaussian mixture models (GMMs) at acoustic modeling for speech recognition on a variety of data sets (Hinton et al., 2012a). In the area of genetics, Xiong et al. use “deep learning” computer algorithms to derive a computational model that takes as input DNA sequences and applies general rules to predict splicing in human tissues (Xiong et al., 2015). It reveals the genetic origins of disease and how strongly genetic variants affect RNA splicing. In the area of natural language understanding, deep learning models have delivered strong results on topic classification, sentiment analysis et al. Sutskever et al. proposed a general approach, the Long Short-Term Memory (LSTM) architecture which can solve the general sequence to sequence problems better than before (Sutskever et al., 2014). In addition, Hinton et al. proposed autoencoder (AE) networks that is an effective way to learn the low-dimensional codes of high-dimensional data. Based on autoencoder, there are also have many excellent works to handle various tasks. Vincent et al. proposed a denoising autoencoder (DAE) which maked the learned representations robust to partial corruption of the input data (Vincent et al., 2008). The denoising autoencoder which initialize the deep architectures layer by layer is very similar with human visual system. Hinton et al. introduced random ‘dropout’ to prevent the overfitting which improve many benchmark tasks and obtain new records for speech and object recognition (Hinton et al., 2012b). Then, Vincent et al. proposed stacked denoising autoencoders (SDAE) which based on stacking layers of stacked denoising autoencoders (Vincent et al., 2010). It is very useful to learn the higher level representations and work well on natural images and handwritten digits. However, for the same reason, they also need a large scale training set and a long training time. They have no advantages to handle the small and middle scale applications.
Moreover, in the field of feature learning models, dimensionality reduction plays a crucial role to handle the problems for compressing, visualizing high-dimensional data and avoiding the “curse of dimensionality” (van der Maaten et al., 2009; van der Maaten, 2007). Traditional dimensionality reduction mainly can be classified into three types: linear or nonlinear, like principal components analysis (PCA) (Jolliffe, 2002) and linearity preserving projection (LPP) (Niyogi, 2004) are linear methods, stochastic neighbor embedding (SNE) (Hinton & Roweis, 2002) is a nonlinear method; supervised or unsupervised, such as marginal Fisher analysis (MFA) (Yan et al., 2007; Zhong et al., 2013) and linear discriminant analysis (LDA) (Fisher, 1936) are supervised methods, PCA is an unsupervised method; local or global, like MFA and SNE are local methods, PCA is a global method. Many feature learning models based on geometry theory provide different solutions to the problem of dimensionality reduction. Yan et al. proposed a general formulation about graph embedding framework can exploit new dimensionality reduction algorithms (Yan et al., 2007). If only directly use some feature learning models to extract the good representation from original data, it often eventually couldn’t get a good outcome. Considering this situation, we try to choose some excellent feature learning models and combine them with some deep learning algorithms. MFA is one special formulation of the graph embedding models based on this framework. It utilizes an intrinsic graph to characterize the intraclass compactness, and another penalty graph to characterize the interclass separability. Our motivation is to combine the advantage of MFA and deep architectures and propose a new initialization method for deep learning algorithms.
There are also have some excellent works about feature learning models combined the deep architectures (Yuan et al.; George et al., 2014; Ngiam et al., 2011). Yuan et al. proposed an improved multilayer learning model to solve the scene recognition task (Yuan et al.). This model overcome the limitation of shallow, one-layer representations for scene recognition. Trigeorgis et al proposed deep Semi-NMF, that is able to learn such hidden representations from different, unknown attributes of a given dataset (George et al., 2014). Ngiam proposed a deep architectures to learn features over multiple modalities (Ngiam et al., 2011). They showed that multi-modality feature learning is better than one modality and achieved good performance on video and audio datasets. However, in general, we can only obtain data from one modality. In this work, we combine the advantages of MFA and deep architectures, which based on stacked feature learning models (Zheng et al., 2014; 2015), then we use some deep learning tricks, like back propagation, denoising and dropout to fine tuning the network. The advantage of this deep architecture is that we can learn the desirable weight matrix even if the training data is not large enough. And compared with traditional deep learning models and shallow feature learning models, our MDA achieved state-of-the-art results in most cases.
3 MARGINAL DEEP ARCHITECTURES (MDA)
In this section, we firstly introduce a novel framework of deep architectures, then we introduce marginal Fisher analysis (MFA) and the proposed marginal deep architectures (MDA) in detail. In addition, we also present some deep learning tricks that we used in the MDA model, including back propagation, denoising and dropout.
3.1 A NOVEL FRAMEWORK OF DEEP ARCHITECTURES
The feature learning problem is generally formulated as follow. Given n data, {xT1 , . . . ,xTn} ∈ <D, where D is the dimensionality of the data space, we seeks the compact representations of these data, i.e., {yT1 , . . . ,yTn } ∈ <d, where d is the dimensionality of the low dimensional embeddings. In order to improve the accuracy of shallow feature learning models, we use stacked feature learning models to construct the deep architectures (Zheng et al., 2014; 2015), which is a general framework for different applications. In this case, the mapping of data from the original D-dimensional space to the resulted d-dimensional space can be described as
D =⇒ D1 =⇒ · · · =⇒ Di =⇒ · · · =⇒ Dp−1 =⇒ d, (1) where D1 is the first higher dimensional space, the number of the node is twice or quadruple as the input layer. Di represents the dimensionality of the i-th intermediate representation space, and p is the total steps of mappings. Here, we can use different feature learning models for the learning of each layer. As the feature learning models are optimized layer by layer, we can obtain the mapping functions between successive layers. The first hidden layer is random by Wr1, and the representation is, a1 = g(WTr1x+ b) (2) where, g(.) is a non-linear activation or transfer function. Then, we can use some feature learning models to initialize the next layers. The representations of next hidden layers are,
ak = g(WTFk−1a k−1 + b) (3)
where, WFk−1 is the weight matrix of the k − 1th layer learned from feature learning models.
3.2 MARGINAL FISHER ANALYSIS (MFA)
Based on our novel framework of deep architecture, we introduce Marginal Fisher Analysis (MFA) to build MDA. Here, many traditional feature learning models, such as linear discriminant analysis
(LDA), can be used as building blocks of MDA. Take LDA as an example. It assumes that the data of each class follow a Gaussian distribution. However, this assumption is not often satisfied in the real world. Without this assumption, LDA can not work well to separate the data with nonlinear structure. Alternatively, MFA can solve this problem effectively. Hence, considering the learning capability, we choose MFA as the build blocks of MDA in our work. MFA used the graph embedding framework to set up an intrinsic graph that characterizes the intraclass compactness and another penalty graph which characterizes the interclass separability. The marginal Fisher criterion is defined as
W∗ = argmin W tr(WTX(D−A)XTW) tr(WTX(Dp −Ap)XTW)
(4)
where D and Dp are diagonal matrices with elements Dii = ∑ j Aij , and D p ij = ∑ j A p ij , respectively. Then we can learn the projection matrix to multiply PCA’s projection and marginal Fisher projection,
WMFA = WPCAW ∗ (5)
3.3 MARGINAL DEEP ARCHITECTURES (MDA)
In order to combine the advantages of MFA and proposed deep architectures, we propose the marginal deep architectures (or MDA). The MDA inherited from the proposed novel framework of deep architectures is shown in Fig. 1. As an input vector x ∈ [0, 1]d, we first map it to higher dimensional space by a random weight matrix Wr1. The representation of first hidden layer is computed as
a1 = s(WTr1x+ b) (6)
where, s(.) is the sigmoid function s(x) = 11+e−x , b is the bias terms, a 1 is the output of first layer. From second layer to (n − 1)-th layer, we use the weight matrices learned from MFA to map layer by layer.
ak = s(WTMFAk−1a k−1 + b) (7)
The last layer is a softmax regression layer and the number of neuron is the number of category. The cost function is defined as,
J(w) = − 1 N ( N∑ i=1 K∑ j=1 I(yi = j) log exp(wTj a n−1 i )∑K l=1 exp(w T l a n−1 i ) ) (8)
where, I(x) is the indicator function, I(x) = 1 if x is true, else I(x) = 0. yi is the label corresponding to xi. Then the probability that xi is classified to j is,
p(yi = j|xi,w) = exp(wTj a n−1 i )∑K
l=1 exp(w T l a n−1 i )
(9)
Taking derivatives, one can show that the gradient is,
∇J(w) = − 1 N N∑ i=1 [xi(I(yi = j)− p(yi = j|xi,w)] (10)
If the n−1 layer’s neurons are more than the last layer, we can continue using MFA to map it. On the contrary, If the n − 1 layer’s neurons are less than last layer, we can randomly initialize the weight matrix between this two layers. Next, in order to improve the MDA, we introduce back propagation, denoising and dropout operation.
3.4 BACK PROPAGATION
In order to adjust the network, we use back propagation (Rumelhart et al., 1986) to compute partial derivative and stochastic gradient descent to update the weight matrixes and the bias terms. For each node i in output layer (n-th layer), we compute an error term as
δni = ∇J(w) (11)
where, J(w) is the cost function computed from Equ.8 and ∇J(w) computed from Equ.10. For each node i in (n− 1)-th to second layer, the error term is computed as,
δki = ( k+1∑ j=1 wkjiδ k+1 j )s ′(zki ) (12)
The back propagation procedure relies on computing the gradient of an objective function with respect to the weights of a multilayer stacked modules. It starting from the output at the top and end to the input at the bottom.
3.5 DENOISING OPERATION
Vincent et al. proposed the denoising autoencoder to improve the robustness of autoencoder (Vincent et al., 2008). It’s very similar with the regularization methods and avoids the “overfitting” problem. The basic idea is to corrupt partial input data by the desired proportion of ν “destruction”. for each input x, a fixed number νd of components are chosen at random, and their value is forced to 0, while the others are left untouched. The initial input x to get a partially destroyed version x̃ by means of a stochastic mapping, x̃ ∼ qD(x̃|x) (13) where, qD(x̃|x) is the unknown distribution. Then, for a hidden representation h,
h = s(WT x̃+ b) (14)
In our MDA, we use this idea to improve the network, please refer to Fig. 1 to find clear sight. For the input layer, the output of first hidden layer is represented as,
a2 = s(WTr1 x̃+ b1) (15)
where, Wr1 is the first layer random weight matrix, b1 is the bias term of first layer. The “denoising” operation is established to a hypothetical additional specific criterion: robustness to partial destruction of the input, which means a good intermediate representation is learned from unknown distribution of its observed input. This operation helps for learning more stable structure and avoids the overfitting problem in most cases.
3.6 DROPOUT
As the same reason with denoising operation, dropout is a trick to prevent overfitting (Hinton et al., 2012b). When a large feedforward neural network is trained on a small training set, dropout performed well on test set. In order to prevent the complex co-adaptations on the training data, the basic idea of dropout is that each hidden node is randomly omitted from the network with a probability of β, so a hidden node can’t rely on other hidden node. In another view, dropout is as a very efficient way of performing model averaging with neural networks. On test set, we train many separate networks and then to apply each of these networks to the test data. Dropout operation can save the train time and then we average the predictions produced by a very large number of different networks. Fig. 1 shows the dropout operation in our MDA.
4 EXPERIMENTS
4.1 DATESET DESCRIPTIONS
We evaluate the performance of MDA on five benchmark data sets. The detail of the data is showed in Tab 1. The USPS 1 data set is a handwritten digits image data set includes 7291 training samples and 2007 test samples from 10 classes with 256 dimensional features. This task is to recognize the digits 0 to 9. The Isolet 2 data set is a collection of audio feature vectors of spoken letters from the English alphabet. It includes 6238 training samples and 1559 test samples from 26 classes with 614 dimensional features. The task is to identify which letter is spoken based on the recorded
1http://www.gaussianprocess.org/gpml/data/ 2http://archive.ics.uci.edu/ml/datasets/ISOLET
(and pre-processed) audio signal. Sensor 3 is a sensorless drive diagnosis data set includes 46816 training samples and 11693 test samples from 11 classes with 48 dimensional features. The features are extracted from electric current drive signals. The task is to classify 11 different classes with different conditions of the drive which has intact and defective components. Covertype 4 contains geological and map-based data from four wilderness areas located in the Roosevelt National Forest of northern Colorado. It includes 15120 training samples and 565892 test samples from 7 classes with 54 dimensional features. The task is to identify forest cover type from cartographic variables. For the IbnSina 5 ancient Arabic document data set, we use 50 pages of the manuscript for training (17543 training samples) and 10 pages for testing (3125 test samples). The data samples belong to 174 classes of subwords and are of dimensionality 200.
In addition, we also use a large scale dataset CIFAR-10 6 to test our MDA on large scale applications. The CIFAR-10 dataset consists of 60000 32 × 32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. We also test our MDA on a specific task which use the CMU motion capture (CMU mocap) data set 7. The CMU mocap data set includes three categories, namely, jumping, running and walking. We choose 49 video sequences from four subjects. For each sequence, the features are generated using Lawrences method 8, with dimensionality 93 (Zhong et al., 2010). By reason of the few samples of CMU, we adopt 10-fold cross-validation in our experiments and use the average error rate and standard deviation to evaluate the performance.
4.2 CLASSIFICATION ON FIVE BENCHMARK DATA SETS
4.2.1 BASELINE METHODS
In order to evaluate the performance of MDA, we compared our MDA with 5 deep learning models include autoencoder (AE) (Hinton & Salakhutdinov, 2006), stacked autoencoders, denoising autoencoders (Vincent et al., 2008), stacked denoising autoencoders (Vincent et al., 2010) and stacked denoising autoencoders with dropout, 2 feature learning models, MFA (Zhong et al., 2013; Yan et al., 2007) and PCA (Jolliffe, 2002), PCA deep architecture base on our uniform framework and the classification accuracy on original space.
4.2.2 EXPERIMENTAL SETTINGS
All of the deep learning methods have the same settings. The size of minibatch was set to 100, the learning rate and momentum were the default value 1 and 0.5, the number of epoch was set to 400, the dropout rate and denoising rate ν were set to 0.1. For the AE and SAE, weight penalty of the L2 norm was set to 10−4. For MFA, the number of nearest neighbors for constructing the intrinsic graph was set to 5, while that for constructing the penalty graph was set to 20. The target spaces of MFA and PCA on different data sets were showed in Tab 1. For the USPS data set, The architecture was set to 256 − 512 − 256 − 128 − 64 − 32. For the Isolet data set ,the architecture was set to 617 − 1324 − 617 − 308. For the Sensor data set, the architecture was set to 48 − 96 − 48 − 24.
3http://archive.ics.uci.edu/ml/datasets/Dataset+for+Sensorless+Drive+Diagnosis# 4http://archive.ics.uci.edu/ml/datasets/Covertype 5http://www.causality.inf.ethz.ch/al data/IBN SINA.html 6http://www.cs.toronto.edu/ kriz/cifar.html 7http://http://mocap.cs.cmu.edu/ 8http://is6.cs.man.ac.uk/∼neill/mocap/
For the Covertype data set, we set the architecture to 54− 216− 108− 54− 27. Finally, for Ibnsina data set, the architecture was set to 200− 400− 200− 100.
4.2.3 CLASSIFICATION RESULTS
The experimental results are shown in Tab. 2. We can see that our MDA achieves the best results on four dataset except the Sensor dataset, but MDA achieves the second best result on Sensor data set and only below the PDA. The PDA achieves the best result on Sensor data set and the second best results on other data sets. These results demonstrate that our uniform deep architectures achieve the good performance in most case. In addition, MDA not only outperform the traditional deep learning models, but also the shallow feature learning models. It shows that our deep architectures based on stacked some feature learning models can learn the better feature than shallow feature learning models.
4.3 EVALUATION
4.3.1 DIFFERENT STRUCTURES FOR MDA
In order to evaluate the desired structures of MDA, we changed the node’s number of the second layer. For USPS data set, we get rid of the second layer and the architecture was 256−128−64−32. Then, we set the number of node of the second layer was as twice as the input layer, the architecture was 256 − 128 − 64 − 32. Next, the number of node was as quadruple as the input layer, the architecture was 256− 1024− 512− 256− 128− 64− 32. Finally, the node’s number is as octuple as the input layer, the architecture was 256 − 2048 − 1024 − 512 − 256 − 128 − 64 − 32. The structures of other data sets are shown in Tab. 3.
The experimental results are shown in Tab. 4. When the the number of nodes of the second layer is as twice as the input layer, MDA achieved the minimum classification error on all data sets except the Covertype data set. When the number of nodes of the second layer is as quadruple as the input
layer, MDA get the worst result on Covertype data set. We can conclude that MDA can work well when the number of nodes of the second layer is as twice or quadruple as the input layer.
4.3.2 DIFFERENT NUMBER OF HIDDEN LAYERS FOR MDA
In order to evaluate how many hidden layers adapt to different datasets, we designed some experiments which have different number of hidden layers. We used 1 ∼ 7 hidden layers on USPS and Isolet datasets and 1 ∼ 5 hidden layers on Covertype, Sensor and Ibnsina datasets. The experimental settings were same as previous experiments.
Tab. 5 shows the classification error on 5 datasets with different hidden layers. All the datasets achieved the best results when hidden layer’s number is 3 except USPS dataset. The USPS dataset achieved the best result when hidden layer’s number is 5. As 1 ∼ 3 hidden layers, with the increase of the number of layers, the classification error is decreasing on all datasets. As small and middle scale applications, we don’t need very deep architectures to handle it. As large scale applications, we can design deeper architectures to achieve better performance.
4.4 CLASSIFICATION ON LARGE SCALE DATASET CIFAR-10
The previous section introduced the advantages of MDA on small and middle scale applications. In order to evaluate the universality of MDA, we chose a relatively large scale dataset CIFAR-10 to test the performance of MDA.
In our experiments, we first transformed the color images to gray images in order to reduce the dimensionality of input. Then we took one sample as a 1024 dimensional vector which is the input of our MDA. So, we can call this data set gray-CIFAR10. The architecture was set to 1024−2048− 1024−512−256−128−64, the minibatch’s size was set to 100, the dropout ratio and denoising ratio were set to 0.1, the number of epoch was set to 400, the learning rate was set to 1, the momentum was set to 0.5. We compared our MDA with previous 6 methods.
Table. 6(a) shows the classification error on gray-CIFAR10, we can see that PDA and MDA achieved the best results in these 7 methods. However, all of the methods on this framework didn’t perform well because we use the gray operation.
4.5 CLASSIFICATION ON CMU MOCAP DATA SET
CMU mocap data set is a very small dataset that only has 49 samples. Traditional deep learning methods didn’t work well in these kind of applications. We test our MDA and PDA and compared them with other 5 deep learning models. The architectures for all deep models (except the PDA) were set to 93− 186− 93− 47− 24. Specially, since the CMU mocap data set only has 49 samples, the PCA method only reduce the dimensionality to 49 at most, so the architecture of PDA was set to
93 − 186 − 24. The denoising ratio and dropout ratio were set to 0.1 on DAE, DAE with dropout, SDAE, SAE, PDA and MDA. The weight penalty on AE was set to 10−4. The learning rate was set to 0.01, the momentum was set to 0.5 and the number of epoch is set to 600. The experiment was test on 10-fold cross validation. The experimental results are shown in Tab. 6(b).
In Tab. 6(b), our PDA and MDA achieved the best results in this dataset and have lower standard deviation than other deep learning models. It demonstrates that our PDA and MDA are more stable than other deep learning models. The traditional autoencoder, SDAE, DAE with dropout achieved the same result in this dataset and better than SAE and DAE.
5 CONCLUSION
In this paper, we proposed a novel deep learning framework that based on stacked some feature learning models to handle small or middle data sets. Then we introduce MFA in this framework, called MDA. The deep learning tricks like backpropagation, denoising and dropout operation are applied on MDA to improve its performance. Extensive experiments on 7 different type data sets demonstrate that MDA performs not only better than shallow feature learning models, but also stateof-the-art deep learning models on small and middle scale applications. The evaluation of MDA show that how to adjust the parameters make the MDA work well. For future work, we plan to try other feature learning models and explore the different structures for this novel deep learning model. In addition, we plan to explore new deep architectures based on this framework to handle the large scale datasets.

The proposed approach consists in a greedy layer wise initialization strategy for a deep MLP model, which is followed by global gradient-descent with dropout for fine-tuning. The initialization strategy uses a first randomly initialized sigmoid layer for dimensionality expansion followed by 2 sigmoid layers whose weights are initialized by Marginal Fisher Analysis (MFA) which learns a linear dimensionality reduction based on a neighborhood graph constructed using class label information (i.e. supervised dimensionality reduction). Output layer is a standard softmax layer.
The approach is thus to be added to a growing list of heuristic layer-wise initialization schemes.
The particular choice of initialization strategy, while reasonable, is not sufficiently well motivated in the paper relative to alternatives, and thus feels rather arbitrary.
The paper lacks clarity in the description of the approach:  MFA is poorly explained with undefined notations (in Eq. 4, what is A? It has not been properly defined); the precise use of alluded denoising in the model is also unclear (is there really training of an additional denoting objective, or just input corruption?).
The question of the (arguably mild) inconsistency of applying a linear dimensionality reduction algorithm, that is trained without any sigmoid, and then passing its learned representation through a sigmoid is not even raised. This, in addition to the fact that sigmoid hidden layers are no longer commonly used (why did you not also consider using RELUs?).
More importantly I suspect methodological problems with the experimental comparisons: the paper mentions using *default* values for learning-rate and momentum, and having (arbitrarily?) fixed epoch to 400 (no early stopping?) and L2 regularization to 1e-4 for some models.
*All* hyper parameters should always be properly hyper-optimized using a validation set (or cross-validation) including early-stopping, and this separately for each model under comparison (ideally also including layer sizes). This is all the more important since you are considering smallish datasets, so that the various initialization strategies act mainly as different indirect regularization schemes: they thus need to be carefully tuned. This casts serious doubts as to the amount of hyper-parameter tuning (close to none?) that went into training the alternative models used for comparison.
The Marginal Fisher Analysis dimensionality reduction initialization strategy may well offer advantages, but as it currently stands this paper doesn’t yet make a sufficiently convincing case for it, nor provide useful insights into the nature of the expected advantages.
I would also suggest, for image inputs such as CIFAR10, to use the qualitative tool of showing the filters (back projected to input space) learned by the different initialization schemes under consideration, as this could help visually gain insight as to what sets methods apart.


One main concern of the deep learning community is to increase the capacity of representation of deep networks by increasing their depth. This requires to scale up the size of the training database accordingly. Indeed a major intuition lies in the fact that the depth of the network and the size of the training set are strongly correlated. However recent works tend to show that deep learning may be handled with smaller dataset as long as the training samples are carefully selected (let us mention for instance curriculum learning). In this context we introduce a scalable and efficient active learning method that can be applied to most neural networks, especially Convolutional Neural Networks (CNN). To the best of our knowledge, this paper is the first of its kind to design an active learning selection scheme based on a variational inference for neural networks. We also deduced a formulation of the posterior and prior distributions of the weights using statistical knowledge on the Maximum Likelihood Estimator. We describe our strategy to come up with our active learning criterion. We assess its consistency by checking the accuracy obtained by successive active learning steps on two benchmark datasets MNIST and USPS. We also demonstrate its scalability towards increasing training set size.
1 INTRODUCTION
A daunting challenge in many contexts is to gather annotated data. This can be a long and tedious process, which often slows down the development of a framework and may jeopardize its economic prospects.
We refer to active learning Cohn (1994) as the field of machine learning which targets building iteratively the annotated training set with the help of an oracle.
In this setting and in a context of pool-based active learning1, a model is trained on a small amount of data (i.e. the initial training set) and a scoring function discriminates samples which should be labeled by the oracle from the ones which do not hold new information for the model. The queried samples are then submitted to an oracle (which can be another decision algorithm for instance in co-training context, or a human expert in interactive learning context) to be labeled. They are then added to the current training set. Finally the model is retrained from scratch. This process is repeated recursively to grow the training set.
Although active learning and deep learning represent two important pillars of machine learning, they have mainly coexisted into independent stream of works owing to the complexity of combining them. The main issues are the scalability and the adaptability of common active learning schemes when considering architectures with a huge number of parameters such as deep networks. Another issue lies in the overall number of training iterations since training a deep architecture remains a computationally expensive process despite all the optimizations through GPU processing. This specificity has prevented deep learning from being prevalent within active learning. Indeed seminal active learning frameworks Cohn (1994) have mainly focused on adding one sample at-a-time. When it comes to selecting a batch of queries, the most intuitive solution is to select top scoring samples.
1Other settings exist but are not considered here for the sake of clarity, we refer the reader to Settles (2012)
Such a solution is immediate in the process but fails to model the correlations between samples. Labeling one sample at-a-time may therefore lead to the labeling of another sample totally useless.
In our work, batches of actively selected samples are added at each training iteration. We propose a batch active learning framework designed for deep architectures, especially Deep Convolutional Neural Networks (CNN).
Batch active learning is highly suitable for deep networks which are trained on minibatches of data at each iterations. Indeed training with minibatches help the training of deep networks, and we empirically noticed that the size of the minibatch is a major hyperparameter. Thus it makes sense to query a batch of unlabelled data whose size would be proportionnal to the size of a minibatch. In our work, batches have the same size as the minibatches but they could be decorrelated by considering importance sampling techniques.
Our model focuses on log loss which is involved in the training process of most neural networks. To achieve the required scalability of active learning for deep architectures, we step away from traditional active learning methods and focus our attention on a more general setting: Maximum Likelihood Estimation (MLE) and Bayesian inference. Provided certain assumptions, our active selection relies on a criterion which is based on Fisher information and is obtained from the minimization of a stochastic method for variational inference. Our active selection relies on the Fisher matrices on the unlabeled data and on the data selected by the active learning step. An approximation of Fisher information, based on a diagonal Kronecker-block decomposition makes our criterion computationally affordable for an active greedy selection scheme.
Variational methods have been previously explored like in Graves (2011) as a tractable approximation to Bayesian inference for Artificial Neural Networks (ANNs). One advantage of such a representation is that it leads to a two-term active learning criterion: one related to the prediction accuracy on the observed data and the second term expressing the model complexity. Such a two-fold criterion is, to the best of our knowledge, the first of the kind scalable for active learning. Such an expression may help both to analyze and to optimize ANNs, not only in an active learning framework but also for curriculum learning.
We dedicate section Related works to the presentation of active learning literature. Section Covering presents the theoretical aspects of our active learning framework, while section Active learning as a greedy selection scheme details our greedy algorithm. We then assess our active learning method through experiments on MNIST and USPS benchmarks. We discuss possible improvements of our approach and connections with previous MLE-based active learning methods before concluding.
2 RELATED WORKS
Active learning is a framework to automatize the selection of instances to be labeled in a learning process. Active learning offers a variety of strategies where the learner actively selects which samples seem “optimal” to annotate, so as to reduce the size of the labeled training set required to achieve equivalent performance.
We consider the context of pool-based active learning where the learner selects its queries among a given unlabeled data set. For other variants (query synthesis, (stream-based) selective sampling) we refer the reader to Settles (2012).
2.1 POOL-BASED ACTIVE LEARNING
When it comes to pool-based active learning, the most intuitive approaches focus on minimizing some error on the target classifier. Uncertainty sampling minimizes the training error by querying unlabeled data on which the current classifier (i.e. from previous training iteration) is assigning labels with the weakest confidence. This method, uncertainty sampling, while being the least computational consuming among all active learning techniques has the main drawback of ignoring much of the output distribution classes, and is prone to querying outliers. Thanks to its low cost and easy setup, uncertainty has been adapted to deep architectures for sentiment classification Zhou et al. (2010). However, deep architectures are subject to adversarial examples, a type of noise we suspect uncertainty selection to be highly sensitive to Szegedy et al. (2014); Goodfellow et al. (2015). Other strategies (expected error reduction, expected output variance reduction) directly minimize the
error on a validation set after querying a new unlabeled sample. However they are computationally expensive especially when considering neural networks.
Traditional active learning techniques handle selection of one sample at-a-time. One of the main drawbacks of the aforementioned active learning techniques is that is does not pay attention to the information held in unlabeled data besides considering it as potential queries. Hence once the strategy for selecting samples to be labeled and added to the training set is defined, the question on the impact of the possible correlation between successive selected samples remains.
To that end, one recent class of methods deals with the selection of a batch of samples during the active process, batch mode active learning. Batch mode active learning selects a set of most informative unlabeled sample instead of a unique sample. Such a strategy is highly suitable when retraining the model is not immediate and require to restart the training from scratch at each iteration as it is the case for neural networks. A simple strategy (whose has also been used for previous deep active learning strategy Zhou et al. (2010)) is to select a batch of top scoring instances. However that strategy fails to consider the correlation among pairs of samples. The redundancy between the so-selected samples may therefore hinder the learning process.
In the context of a deep learning scenario, if several elements related to the same direction of the gradient are in the same minibatch, the gradient descent in the next learning step may lead at once too close to a local minimum, diverting the process away from the global minimum.
While one sample at-a-time can prevent from being misled that way, it gets prohibitive when considering big data because the number of iterations is equal to the number of learning samples, unlike the batch-based strategies.
Recently some solutions have been proposed for choosing an appropriate subset of samples so as to minimize any significant loss in performance. Those methods consider the minimization of the Kullback-Leibler (KL) divergence between the resampled distribution of any possible subset selected for the active process and the whole unlabeled data set distribution. A lower bound of the negative of this KL divergence is then defined and rewritten as a submodular function. The minimization of the initial KL divergence becomes then a problem of submodular maximization Hoi et al. (2006). In Wei et al. (2015), Wei et al have designed several submodular functions so as to answer at best the need of specific classifiers (Naive Bayes Classifiers, Logistic Regression Classifier, Nearest Neighbor Classifier). However, their approach is hardly scalable to handle all the information from non-shallow classifiers such as deep networks.
Another solution to minimize the correlation among a set of queries is to perform bayesian inference on the weights of a neural network. In a bayesian context, a neural network is considered as a parametric model which assigns a conditional probability on the observed labeled data A given a set of weights w. The weights follow some prior distribution P (α) depending on the parameter α and the posterior distribution of the weights P (w | A, α) is deduced. The goal is thus to maximize the posterior probability of the weights on the observed data A. Indeed bayesian inference expresses the uncertainty of the weights which consequently leads to a relevant exploration of the underlying distribution of the input data X. When it comes to active learning, the learner needs not only to estimate the posterior given the observed data A but also to consider the impact of new data on that posterior Golovin et al. (2010). In our context, bayesian inference is intractable, partially due to the high number of weights involved in a deep network. To solve this issue, Graves (2011) introduced a variational approximation to perform bayesian inference on neural networks. Specifically he approximates the posterior distribution P (w | A, α) with a tractable distribution of his choice Q(w | β) depending on a new parameter β. The quality of the approximation Q(w | β) compared to the true posterior P (w | A, α) is measured by the variational free energy F with respect to the parameters α and β. F has no upper bound but gets closer to zero as both distributions become more and more similar.
F(α, β) = −Ew∼Q(β)
( ln ( P (A | w)P (w | α)
Q(w | β)
)) (1)
Finally in Graves (2011), F is then expressed as a minimum description loss function on α and β: F(α, β) = Ew∼Q(β)(L(A;w)) +KL(Q(w) || P(α)) (2)
where KL is the Kullback Leibler divergence between Q(β) and P(α).
Under certain assumptions on the family distribution for the posterior and prior of the weights ( diagonal gaussian ...), Graves proposed a backpropagation compatible algorithm to train an ensemble of networks, whose weights are sampled from a shared probability distribution.
The primary purpose of the variational free energy is to propose a new training objectif for neural network by learning α and β. In an active learning context, the main drawback of variational free energy based method is that it requires to update by backpropagation the parameters for each unlabeled data submitted as a query. However we know from statistical assumption on the maximum likelihood the posterior and prior distribution of trained weights given the current labeled training set: if and only if we consider trained networks, we know how to build α and β in a unique iteration without backpropagation. This knowledge helps us to extend Graves first objectives to the use of variational free energy to measure how new observations affect the posterior.
3 COVERING
3.1 VARIATIONAL INFERENCE ON NEURAL NETWORK WITH GAUSSIAN POSTERIOR WEIGHT DISTRIBUTION
As done for the majority of neural networks, we measure the error of the weights w by the negative log likelihood on an observed set of annotated data A:
L(A;w) = − ∑
(x,y)∈A
ln(P (y | x,w)) (3)
We consider the Maximum Likelihood Estimator (MLE) W as the value which makes the data observed A as likely as possible for a fixed architecture. Note than even for a fixed A in the case of neural network,W may not be unique.
W = argminwL(A;w) (4)
When assuming that an arbitrary parameterW∗ is governing the data generation process, we know that the expected negative log likelihood is lower bounded by the expected negative log likelihood of the true parameterW∗ governing the data generation process. What it means is that no distribution describes the data as well as the true distribution that generated it. It turns out that, under certain assumptions, we can prove using the central limit theorem that the MLE is asymptotically normal with a mean equal to the true parameter value and variance equal to the inverse of the expected Fisher information evaluated at the true parameter.
If we denote by X the underlying data distribution,WX the MLE andW∗X the true parameter, we know thatWX is a sample from a multivariate gaussian distribution parametrized byW∗X . Note that in this context we assume the Fisher matrices are invertible.
WX ∼ N (W∗X , I−1X (W ∗ X)) (5)
However the expected Fisher information on the underlying distribution is intractable. Eventually, using the law of large numbers, we know that the observed Fisher information converges to the expected Fisher information as the sample size increases. Another theoretical limitation is that the true parameter is unknown but we can approximate its observed Fisher information with the observed Fisher information at the MLE because of the consistency of the MLE. For a sake of simplicity we keep the same notation for observed and expected Fisher matrix.
Let denote by Y the random variable after resampling the underlying distribution X using an active learning strategy. W∗X,W∗Y are the true parameters with respect to their respective data distributions and their respective MLE variablesWX,WY, then the following relations hold:
WX ∼ N (W∗X, I−1X (W ∗ X)) WY ∼ N (W∗Y, I−1Y (W ∗ Y))
(6)
We thus notice than in an active learning context, the learner is trained on data uniformly sampled from Y, while the optimal solution would be when training on data uniformly sampled from X.
The asymptotic behaviour provides us with a prior distribution of the weights based on the data distribution X. In our context of active learning, we approximate the posterior distribution with the MLE distribution induced by the resampling Y. Hence we define a prior and posterior distribution which did not require to be learnt by backpropagation directly but depend on the two data distribution X and Y.
P (α) ≡ P (αX) = N (W∗X, I−1X (W ∗ X)) Q(β) ≡ Q(βY) = N (W∗Y, I−1Y (W ∗ Y))
(7)
Our active learning scheme relies on the selection of input data whose induced MLE distribution Q(βY) is minimizing the variational free energy.
Y = argminY F(αX, βY) (8)
It consists in the minimization of the sum of two terms which we denote respectively by the training factor EW∼Q(β)(L(A;W)) and the generalization factor KL(Q(β) || P(α)). It is possible to analyze both terms independently and explain their role into the minimization:
• Training factor: Ideally, the Cramer Rao bound implies that the minimum on the training factor is reached whenQ(β) matches the asymptotically most efficient estimator of the optimal parameter on the error loss on the observed data. Hence the training factor corresponds to the minimization of the error on the observed data A.
• Generalization factor: Empirical results Choromanska et al. (2015) tend to show that the variance of the accuracy diminishes as the depth of the classifier increases. So our ultimate goal would be to converge to any set of parameters of the MLE distribution as their effectiveness is similar. The goal of the generalization factor is to converge to the asymptotic distribution on the whole input distribution X and to minimize the error of prediction of new input data.
If the expression of the variational free energy provides us a theoretical context to work on, the usage of Fisher matrices of deep networks renders it computationally unaffordable. Indeed the Fisher matrix is a quadratic matrix in terms of the number of parameters of the deep networks. Because of the huge number of parameters involved, such matrices takes a lot of memory and processing them costs a lot of ressources, especially if the operations may be repeated often in the framework (as it would be the case for every possible query processed by an active learning scheme.)
The next section 3.2 explains the different approximation proposed to deduce a more friendly user criterion.
3.2 APPROXIMATIONS
3.2.1 KRONECKER FACTORED APPROXIMATION OF THE FISHER INFORMATION
OF A CNN
Recently an approximation of the Fisher information for deep architectures has been proposed first for fully connected layer in Martens & Grosse (2015), and then for convolutional layer as well in Grosse & Martens (2016). The block kronecker decomposition content (ψ, τ ) is explained in Martens & Grosse (2015); Grosse & Martens (2016)
Based on their decomposition definition, we define the evaluation of blocks of the Fisher information at a certain point xi(ψxi,l, τxi,l) and an empirical estimation of the Fisher matrix on a set of data A. A sum up of their decomposition is presented in Eq. (9) while the exact content of the kronecker
blocks ψ and τ is left as undescribed in this paper for the sake of concision.
IA(W) = diag([ψA,l(W)⊗ τA,l(W)]Ll=1)
ψA,l(W) = 1 | A | ∑ xi∈A ψxi,l(W)
τA,l(W) = 1 | A | ∑ xi∈A τxi,l(W)
(9)
The strength of this decomposition lies in the properties of block diagonal combined with those of the kronecker product. ψ and τ are respectively related to the covariance matrix of the activation and the covariance of the derivative given the input of a layer. Recent deep architectures tend to prevail the depth over the width (i.e. the number of input and output neurons) so this expression becomes really suitable and tractable.
3.3 APPROXIMATION OF THE TRAINING FACTOR
Despite the block kronecker product approximation of the Fisher matrix, sampling on Q(β) requires to compute the inverse. Because the kronecker blocks may still have an important number of parameters involved (especially the first fully connected layer suceeding to a convolutional layer), the inverse of the blocks may be still too computationally expensive. To approximate the training factor, we opt for a second order approximation of the log likelihood for parametersW close to the mean parameterW∗Y of Q(β).
L(A;W) ≈ L(A;W∗Y) + ∂L(A;W∗Y)
∂W + (W −W∗Y)T ∂2L(A;W∗Y) ∂W ′∂W (W −W∗Y) (10)
Our first approximation consists in assuming that the MLE parameter ŵ of the currently trained network is a good approximator ofW∗Y. Because the network has converged on the current set of observed data A the first derivative of the log likelihood is also set to zero. Hence Eq. (10) thus becomes:
L(A;W) ≈ L(A; ŵ) + (W − ŵ)T I−1A (ŵ)(W − ŵ) (11)
To compute the expectation over the range of weights sampled fromQ(β) we need to upperbound the expectation of the dot product ofW given the Fisher matrix. Because we assume our Fisher matrices invertible, and because a covariance matrix is at least semi-definite, our Fisher matrices are positive definite matrix. Hence every eigenvalue is positive and the trace of the Fisher matrix is greater than its maximum eigenvalue. From basic properties of the variance covariance matrix, if we denote by N the number of parameters in the network we obtain the following upperbound for the training factor:
EW∼Q(β)(L(A;W)) ≤ L(A; ŵ) + N√ π Tr(I−1Y (ŵ) T I−1A (ŵ)I −1 Y (ŵ)) (12)
When it comes to the trace of the inverse, we approximate it by the closest lower bound with the inverse of the trace like in Wei et al. (2015).
EW∼Q(β)(L(A;W)) ∝∼ L(A; ŵ) + N√ π
N2
Tr(IY(ŵ)IA(ŵ)IY(ŵ)T ) (13)
3.4 APPROXIMATION OF THE GENERALIZATION FACTOR
Our generalization factor corresponds to the KL divergence between the approximation of our posterior Q(β) and the prior P(α). Because both distributions are multivariate gaussians, we have a direct formulation of the KL which is always definite since the Fisher matrices are invertible:
KL(Q(β) || P(α)) = 1 2
( ln ( det(I−1X (W∗X)) det(I−1Y (W∗Y)) ) −N+Tr(IX(W∗X)I−1Y (W ∗ Y))+(W∗X−W∗Y)T IX(W∗X)(W∗X−W∗Y) ) (14)
Our first approximation consists in assuming that the MLE parameter ŵ of the currently trained network is a good approximator of both optimal parametersW∗X,W∗Y like in Zhang & Oles (2000). We also upper bound the determinant with a function of the trace and the number N of parameters. When it comes to the trace of the inverse, we approximate it again by the closest lower bound with the inverse of the trace.
KL(Q(β) || P(α)) ∝∼ N
( ln ( Tr(IY(ŵ)I−1X (ŵ)) ) +
N
Tr(IY(ŵ)I−1X (ŵ))
) (15)
3.5 APPROXIMATION OF THE VARIATIONAL FREE ENERGY
In the previous subsections, we proposed independent approximations of both our sub-criteria: the training factor and the generalization factor. However the scale of our approximations may not be balanced so we sum up our criterion with an hyperparameter factor γ which counterparts the difference of scale between the factors:
F ∝∼ γ ( N√ π
N2
Tr(IY(ŵ)IA(ŵ)IY(ŵ)T )
) +N ( ln ( Tr ( IY(ŵ)I−1X (ŵ) )) +
N
Tr(IY(ŵ)I−1X (ŵ)) ) (16)
We approximate the expected Fisher matrices on the underlying distribution Y and X by the observed Fisher matrices on a set of data sampled from those distributions. This approximation is relevant due to the consistenty of the MLE.
As we are in a pool-based selection case, we dispose at first of two sets of data: A and U which denote respectively the annotated observed data and unlabeled data. Note that the derivatives in the Fisher matrix computation implies to know the label of the samples. Thus at each active learning step, an unknown label is approximated by its prediction from the current trained network. We denote by S the subset of data to be queried by an oracle. The size of S is fixed with | S |= K. S is the subset sampled from Y while U is sampled from X. Finally an approximation of F will be:
F ∝∼ γ ( N√ π
N2
Tr(IS(ŵ)IA(ŵ)IS(ŵ)T )
) +N ( ln ( Tr ( IS(ŵ)I−1U (ŵ) )) +
N
Tr(IS(ŵ)I−1U (ŵ)) ) (17)
Now we express the trace based on the approximation of the Fisher matrix: we consider that every Fisher matrix for CNN is a L diagonal block matrix, with L the number of layers of the CNN. Every block is made of a kronecker product of two terms ψ and τ . We rely on the properties involved by the choice of this specific matrix topology to obtain a more computationally compliant approximation of F in Eq. (18):
F ∝∼γ ( N√ π N2∑ l∈(1,L) Tr(ψS,l(ŵ)ψ −1 A,l(ŵ)ψS,l(ŵ) T )Tr(τS,l(ŵ)τ −1 A,l(ŵ)τS,l(ŵ) T ) )
+N ( ln ( ∑ l∈(1,L) Tr ( ψS,l(ŵ)ψ −1 U,l(ŵ) ) Tr(τS,l(ŵ)τ −1 U,l (ŵ)) )
+ N∑
l∈(1,L) Tr(ψS,l(ŵ)ψ −1 U,l(ŵ))Tr(τS,l(ŵ)τ −1 U,l (ŵ))
) (18)
4 ACTIVE LEARNING AS A GREEDY SELECTION SCHEME ON THE VARIATIATIONAL FREE ENERGY
The selected subset S selected at one step of active learning is only involved through the kronecker product of the Fisher matrix IS(ŵ). We express our approximation of the free energy by a criterion
on the subset S in Eq. (19): C(S;A,U) =γ ( N√ π N2∑ l∈(1,L) Tr(ψS,l(ŵ)ψA,l(ŵ)ψS,l(ŵ) T )Tr(τS,l(ŵ)τA,l(ŵ)τS,l(ŵ)T ) )
+N ( ln ( ∑ l∈(1,L) Tr(ψS,l(ŵ)ψ −1 U,l(ŵ))Tr(τS,l(ŵ)τ −1 U,l (ŵ)) )
+ N∑
l∈(1,L) Tr(ψS,l(ŵ)ψ −1 U,l(ŵ))Tr(τS,l(ŵ)τ −1 U,l (ŵ)) ) (19)
Finally we estimate our subset S by a greedy procedure: to be more robust to outliers and for reasons of computational efficiency, we select first a pool of samples D ⊂ U which we will use as the set of possible queries. We recursively build S ⊂ D by picking the next sample xi ∈ D which minimizes C(S ∪ {xi};A,U) among all remaining samples in D. When it comes to the training factor coefficient, we notice that it is a quadratic term in IS(ŵ) which increases the complexity in a greedy selection scheme. Our choice is to estimate the trace in the following way:
Tr(ψS∪{x},l(ŵ)ψA,l(ŵ)ψS∪{x},l(ŵ) T ) ≈ Tr(ψS,l(ŵ)ψA,l(ŵ)ψS,l(ŵ)T )+Tr(ψ{x},l(ŵ)ψA,l(ŵ)ψ{x},l(ŵ)T )
Pseudo-code and illustration of the algorithm are provided in table 1 in appendix.
5 EXPERIMENTS
We demonstrate the validity of our approach on two datasets: MNIST (28-by-28 pictures, 50.000 training samples, 10.0000 validation samples and 10.000 test samples) and USPS (16-by-16 pictures, 4185 training samples, 464 validation samples and 4649 testing samples) both gray scaled digits image datasets. We describe the CNN configuration and the hyperparameters settings in table 2 in appendix. Note that we do not optimize the hyperparameters specifically for the size of the current annotated training set A. We picked those two similar datasets to judge of the robustness of our method against different size of unlabeled datasets, as expected our method is efficient on both small and large databases.
5.1 TEST ERROR
We run 10 runs of experiments and average the error on the test set of the best validation error before a pass of active learning. We start from an annotated training set of the size of one minibatch selected randomly. We stop both set of experiments after 30% of the training set has been selected (15.000 image for MNIST, 1255 for USPS). We compare the lowest test error achieved so far by our MLE based method against naive baselines: uncertainty sampling, curriculum sampling and a random selection of a minibatch of examples. We measure both uncertainty and curriculum scores based on the log likelihood of a sample using as label its prediction on the full network. While uncertainty selects samples with the highest log likelihood, our version of curriculum does the exact contrary. We select randomly the set of possible queries D among the unlabeled training data. Its size is set to 30 times the minibatch size. We present the results in two phases for the sake of clarity in figure 1 for MNIST and figure 2 for USPS: the first rounds of active learning when the annotated training set is almost empty, and the second round which is more stable in the evolution of the error. In both phases and for both databases we observe a clear difference between the test error achieved by our MLE method with the test error obtained by selecting randomly the data to be queried. Moreover the error achieved by our method on 30 % is close (even equivalent in the case of USPS), to the error achieved using the standard full training sets defined for both datasets (this error rate is defined as yellow line groundtruth on the figures). The experiments made appear that curriculum learning is not a good active learning strategy for both tested datasets. As for the uncertainty selection, it works really well on MNIST while it fails on USPS. While MNIST is a pretty clean database, USPS contains more outliers and noisy samples rendering it more difficult in terms of accuracy even though both databases are designed to assess digit classification. As other works we mentioned in the related work section, we are led to explain uncertainty selection to select useless samples with the amount of outliers and noisy samples in USPS.
5.2 TIME COMPLEXITY
To validate our method in terms of scalability and time complexity, we measured in seconds, the current processor time for one pass of active learning. We repeated this evaluation for different size of query (8 to 128 unlabeled samples added to the currrent training set). For this experiments we used a laptop with a Titan-X (GTX 980 M) with 8 GB RAM GPU memory. Metrics were reported in figure 3. Our criterions takes few seconds to select a batch of query of hundreds of unlabeled data. Moreover the evolution of the time given the size of the query is less than linear.
Page 1
9
6 DISCUSSION
We believe our method is a proof of concept for the use of variational inference for active learning on deep neural networks. However our approximations are subject to improvements which may lead to faster convergence and lower generalization error.
The first point to raise is that our approximation of the posterior is an asymptotic distribution which may be unstable on a small subset of observed data, as it is the case for active learning. Such a distribution may be regularized by taking the probability provided by the central limit theorem about how well our data fits to the asymptotic gaussian distribution. When it comes to the KFAC approximation, it suffers from the same issue and could be regularized when evaluating on small subset. A refinement of the approximations, especially for the generalization factor, following the approaches of submodular functions may be investigated.
Finally, an interesting observation is that our formulation of the variational free energy finds similarities with other MLE based active learning criteria previously proposed in the litterature. Indeed, in Zhang & Oles (2000) the authors study active learning by looking among the possible resampling of the input distribution. They formulate their criterion as the minimization of the trace of the inverse Fisher of the resampled distribution multiplied by the Fisher matrix on the input distribution: minS Tr(I−1S (ŵ)IU (ŵ))
7 CONCLUSION
In a nutshell, we proposed a scalable batch active learning framework for deep networks relying on a variational approximation to perform bayesian inference. We deduced a formulation of the posterior and prior distributions of the weights using statistical knowledge on the Maximum Likelihood Estimator. Those assumptions combined with an existing approximation of the Fisher information for neural network, lead us to a backpropagation free active criterion. Eventually we used our own approximations to obtain a greedy active selection scheme.
Our criterion is the first of the kind to scale batch active learning to deep networks, especially Convolutional Neural Networks. On different databases, it achieves better test accuracy than random sampling, and is scalable with increasing size of queries. It achieves near optimal error on the test set using a limited percentage (30%) of the annotated training set on larger and more reduced dataset. Our works demonstrated the validity of batch mode active learning for deep networks and the promise of the KFAC approximations for deep Fisher matrices for the active learning community. Such a solution is also interesting as a new technique for curriculum learning approach.

Quality:
The paper initiates a framework to incorporate active learning into the deep learning framework, mainly addressing challenges such as scalability that accompanies the training of a deep neural network.
However, I think the paper is not well polished; there are quite a lot of grammatical and typing errors.
Clarity:
The paper needs major improvements in terms of clarity. The motivations in the introduction, i.e., why it is difficult to do active learning in deep architectures, could be better explained, and tied to the explanation in Section 3 of the paper. For example, the authors motivated the need of (mini)batch label queries, but never mention it again in Section 3, when they describe their main methodology.
The related work section, although appearing systematic and thorough, is a little detached from the main body of the paper (related work section should not be a survey of the literature, but help readers locate your work in the relevant literature, and highlight the pros and cons. In this perspective, maybe the authors could shorten some explanations over the related work that are not directly related, while spending more time on discussing/comparing with works that are most related to your current work, e.g., that of Graves '11.
Originality & Significance:
The authors proposed an active learning training framework. The idea is to treat the network parameter optimization problem as a Bayesian inference problem (which is proposed previously by Graves) and formulate the active learning problem as that of sampling the most informative data, where the informativeness is defined by the variational free energy, which depends on the Fisher information. To reconcile the computational burden of computing the inverse of Fisher Information matrix, the authors proposed techniques to approximate it (which seems to be novel)
I think that this paper initiates an interesting direction: one that adapts deep learning to label-expensive problems, via active learning. But the paper needs to be improved in terms of presentation.