In recent years, many deep architectures have been proposed in different fields. However, to obtain good results, most of the previous deep models need a large number of training data. In this paper, for small and middle scale applications, we propose a novel deep learning framework based on stacked feature learning models. Particularly, we stack marginal Fisher analysis (MFA) layer by layer for the initialization of the deep architecture and call it “Marginal Deep Architectures” (MDA). In the implementation of MDA, the weight matrices of MFA are first learned layer by layer, and then we exploit some deep learning techniques, such as back propagation, dropout and denoising to fine tune the network. To evaluate the effectiveness of MDA, we have compared it with some feature learning methods and deep learning models on 7 small and middle scale real-world applications, including handwritten digits recognition, speech recognition, historical document understanding, image classification, action recognition and so on. Extensive experiments demonstrate that MDA performs not only better than shallow feature learning models, but also state-of-the-art deep learning models in these applications.
1 INTRODUCTION
Deep learning methods have achieved desirable performance in many domains, such as image classification and detection, document analysis and recognition, natural language processing, video analysis (Krizhevsky et al., 2012; Chan et al., 2014; Ciresan et al., 2010; Collobert & Weston, 2008; Le et al., 2011). Deep learning methods learn the data representation by using multiple processing layers, which discover the intricate structure of high dimensional data with multiple levels of abstraction (LeCun et al., 2015). For example, for face recognition, the learned features of first layer may be the edges, directions and some local information. The second layer typically detects some object parts which are combination of edges and directions. The higher layers may further abstract the face image by combining the features of previous layers (outline of the eyes, nose, lips). This procedure is very similar with human visual and perceptual system.
In recently years, many deep learning methods have been proposed (l. Boureau & others, 2008; Lee et al., 2009b;a; Hinton & Salakhutdinov, 2006). However, most models meet some difficult problems to solve, such as some parameters need to be randomly initialized, like the weight matrix of two successive layers in deep belief networks (DBNs) and the convolution kernel in convolutional neural networks (CNNs). In addition, traditional deep learning methods need a large scale training data to train the complex networks. It causes many problems in the training process. If we don’t initialize the parameters properly, the optimization procedure might need a long training time and fall into local minima. Alternatively, many feature learning models have been proposed to learn the intrinsic structures of high-dimensional data and avoid the curse of dimensionality. In particular, most of them can be trained with small and middle scale of data and their learning algorithms are generally based on closed-form solution or convex optimization. For instance, marginal Fisher analysis (MFA) (Yan et al., 2007; Zhong et al., 2013) is one of the feature learning models that is a supervised method based on the graph embedding framework. It utilizes an intrinsic graph to characterize the intraclass compactness, and another penalty graph to characterize the interclass separability. Its optimal solution can be learned by generalized eigenvalue decomposition. However,
on the one hand, shallow feature learning models cannot work well on the data with highly nonlinear structure; on the other hand, few efforts are made to combine shallow feature learning models for the design of deep architectures.
In order to simultaneously solve the existing problems in deep learning methods and combine the advantages of feature learning models, we proposed a novel deep learning method based on stacked feature learning models. Particularly, we stack marginal Fisher analysis (MFA) layer by layer for the initialization of the deep architecture and call it “Marginal Deep Architectures” (MDA). Firstly, the input data are mapped to higher dimensional space by using random weight matrix. Then we use MFA to learn the lower dimensional representation layer by layer. In the implementation of this architecture, we add some tricks in the training process, such as back propagation, dropout and denoising to fine tune the network. Finally, the softmax layer is connected to the last feature layer. We have compared our MDA with some feature learning methods and deep learning models on different domains of datasets (including handwritten digits recognition, speech recognition, historical document understanding, image classification, action recognition and so on). Extensive experiments demonstrate that MDA performs not only better than shallow feature learning models, but also state-of-the-art deep learning models in small and middle scale applications.
The contributions of this work are highlighted as follows.
1. We propose a novel structure to build a deep architecture. The first hidden layer has twice or quadruple neurons as the input layer. Then we can use some feature learning models layer by layer to learn the compact representations of data. Finally, we set the last layer as a softmax classifier.
2. Traditional deep learning models in general need a large scale training data. Compared with traditional deep learning models, MDA can work better than traditional deep learning models in small and middle scale applications because the initialization of the weight matrices using MFA is much better than that using random initialization.
3. Our MDA can work well in different domains of datasets, such as handwritten digits, spoken letters and natural images. Extensive experiments demonstrate that MDA is a general model to handel small and middle scale data. On the other hand, for large scale datasets, like CIFAR-10, MDA works comparatively with other deep learning methods.
The rest of this paper is organized as follows: In Section 2, we give a brief overview of related work. In Section 3, we present the marginal Fisher analysis (MFA) and the proposed marginal deep architectures (MDA) in detail. The experimental settings and results are reported in Section 4, while Section 5 concludes this paper with remarks and future work.
2 RELATED WORK
With the development of deep learning methods, many deep networks have been proposed in recent years (Donahue et al., 2013; Krizhevsky et al., 2012; Long et al., 2015; Zhou et al., 2014). These deep learning models show their powerful performance in various fields, such as image classification and analysis, document analysis and recognition, natural language processing et al. In the area of image analysis, Hinton et al. proposed a large, deep convolutional neural network (Alex net) to classify the 1.2 million high-resolution images in the ImageNet. It uses efficient GPU to speed their method. The results show that a large, deep convolutional neural network is capable of achieving recordbreaking results on a highly challenging dataset using purely supervised learning (Krizhevsky et al., 2012). In order to popularize the deep convolutional neural network, Donahue ea al. proposed DeCAF (Deep Convolutional Activation Feature) which is trained in a fully supervised fashion on a large, fixed set of object recognition tasks (Donahue et al., 2013). DeCAF provides a uniform framework for researchers who can improve and change this framework on some specific tasks. However, its performance at scene recognition has not attained the same level of success. In order to handle this problem, Zhou et al. introduce a new scene-centric database called Places with over 7 million labeled pictures of scenes. Then, they learn the deep features for scene recognition tasks by using the same architecture as ImageNet, and establish new state-of-the-art results on several scenecentric datasets (Zhou et al., 2014). However, these methods based on convolutional operation need very large scale training samples and a long training time. They can not work well on small and middle scale applications.
In other domains, deep learning methods also achieve good performance. Hinton et al. represent the shared views of four research groups that have had recent successes in using DNNs for automatic speech recognition (ASR). The DNNs that contain many layers of nonlinear hidden units and a very large output layer can outperform Gaussian mixture models (GMMs) at acoustic modeling for speech recognition on a variety of data sets (Hinton et al., 2012a). In the area of genetics, Xiong et al. use “deep learning” computer algorithms to derive a computational model that takes as input DNA sequences and applies general rules to predict splicing in human tissues (Xiong et al., 2015). It reveals the genetic origins of disease and how strongly genetic variants affect RNA splicing. In the area of natural language understanding, deep learning models have delivered strong results on topic classification, sentiment analysis et al. Sutskever et al. proposed a general approach, the Long Short-Term Memory (LSTM) architecture which can solve the general sequence to sequence problems better than before (Sutskever et al., 2014). In addition, Hinton et al. proposed autoencoder (AE) networks that is an effective way to learn the low-dimensional codes of high-dimensional data. Based on autoencoder, there are also have many excellent works to handle various tasks. Vincent et al. proposed a denoising autoencoder (DAE) which maked the learned representations robust to partial corruption of the input data (Vincent et al., 2008). The denoising autoencoder which initialize the deep architectures layer by layer is very similar with human visual system. Hinton et al. introduced random ‘dropout’ to prevent the overfitting which improve many benchmark tasks and obtain new records for speech and object recognition (Hinton et al., 2012b). Then, Vincent et al. proposed stacked denoising autoencoders (SDAE) which based on stacking layers of stacked denoising autoencoders (Vincent et al., 2010). It is very useful to learn the higher level representations and work well on natural images and handwritten digits. However, for the same reason, they also need a large scale training set and a long training time. They have no advantages to handle the small and middle scale applications.
Moreover, in the field of feature learning models, dimensionality reduction plays a crucial role to handle the problems for compressing, visualizing high-dimensional data and avoiding the “curse of dimensionality” (van der Maaten et al., 2009; van der Maaten, 2007). Traditional dimensionality reduction mainly can be classified into three types: linear or nonlinear, like principal components analysis (PCA) (Jolliffe, 2002) and linearity preserving projection (LPP) (Niyogi, 2004) are linear methods, stochastic neighbor embedding (SNE) (Hinton & Roweis, 2002) is a nonlinear method; supervised or unsupervised, such as marginal Fisher analysis (MFA) (Yan et al., 2007; Zhong et al., 2013) and linear discriminant analysis (LDA) (Fisher, 1936) are supervised methods, PCA is an unsupervised method; local or global, like MFA and SNE are local methods, PCA is a global method. Many feature learning models based on geometry theory provide different solutions to the problem of dimensionality reduction. Yan et al. proposed a general formulation about graph embedding framework can exploit new dimensionality reduction algorithms (Yan et al., 2007). If only directly use some feature learning models to extract the good representation from original data, it often eventually couldn’t get a good outcome. Considering this situation, we try to choose some excellent feature learning models and combine them with some deep learning algorithms. MFA is one special formulation of the graph embedding models based on this framework. It utilizes an intrinsic graph to characterize the intraclass compactness, and another penalty graph to characterize the interclass separability. Our motivation is to combine the advantage of MFA and deep architectures and propose a new initialization method for deep learning algorithms.
There are also have some excellent works about feature learning models combined the deep architectures (Yuan et al.; George et al., 2014; Ngiam et al., 2011). Yuan et al. proposed an improved multilayer learning model to solve the scene recognition task (Yuan et al.). This model overcome the limitation of shallow, one-layer representations for scene recognition. Trigeorgis et al proposed deep Semi-NMF, that is able to learn such hidden representations from different, unknown attributes of a given dataset (George et al., 2014). Ngiam proposed a deep architectures to learn features over multiple modalities (Ngiam et al., 2011). They showed that multi-modality feature learning is better than one modality and achieved good performance on video and audio datasets. However, in general, we can only obtain data from one modality. In this work, we combine the advantages of MFA and deep architectures, which based on stacked feature learning models (Zheng et al., 2014; 2015), then we use some deep learning tricks, like back propagation, denoising and dropout to fine tuning the network. The advantage of this deep architecture is that we can learn the desirable weight matrix even if the training data is not large enough. And compared with traditional deep learning models and shallow feature learning models, our MDA achieved state-of-the-art results in most cases.
3 MARGINAL DEEP ARCHITECTURES (MDA)
In this section, we firstly introduce a novel framework of deep architectures, then we introduce marginal Fisher analysis (MFA) and the proposed marginal deep architectures (MDA) in detail. In addition, we also present some deep learning tricks that we used in the MDA model, including back propagation, denoising and dropout.
3.1 A NOVEL FRAMEWORK OF DEEP ARCHITECTURES
The feature learning problem is generally formulated as follow. Given n data, {xT1 , . . . ,xTn} ∈ <D, where D is the dimensionality of the data space, we seeks the compact representations of these data, i.e., {yT1 , . . . ,yTn } ∈ <d, where d is the dimensionality of the low dimensional embeddings. In order to improve the accuracy of shallow feature learning models, we use stacked feature learning models to construct the deep architectures (Zheng et al., 2014; 2015), which is a general framework for different applications. In this case, the mapping of data from the original D-dimensional space to the resulted d-dimensional space can be described as
D =⇒ D1 =⇒ · · · =⇒ Di =⇒ · · · =⇒ Dp−1 =⇒ d, (1) where D1 is the first higher dimensional space, the number of the node is twice or quadruple as the input layer. Di represents the dimensionality of the i-th intermediate representation space, and p is the total steps of mappings. Here, we can use different feature learning models for the learning of each layer. As the feature learning models are optimized layer by layer, we can obtain the mapping functions between successive layers. The first hidden layer is random by Wr1, and the representation is, a1 = g(WTr1x+ b) (2) where, g(.) is a non-linear activation or transfer function. Then, we can use some feature learning models to initialize the next layers. The representations of next hidden layers are,
ak = g(WTFk−1a k−1 + b) (3)
where, WFk−1 is the weight matrix of the k − 1th layer learned from feature learning models.
3.2 MARGINAL FISHER ANALYSIS (MFA)
Based on our novel framework of deep architecture, we introduce Marginal Fisher Analysis (MFA) to build MDA. Here, many traditional feature learning models, such as linear discriminant analysis
(LDA), can be used as building blocks of MDA. Take LDA as an example. It assumes that the data of each class follow a Gaussian distribution. However, this assumption is not often satisfied in the real world. Without this assumption, LDA can not work well to separate the data with nonlinear structure. Alternatively, MFA can solve this problem effectively. Hence, considering the learning capability, we choose MFA as the build blocks of MDA in our work. MFA used the graph embedding framework to set up an intrinsic graph that characterizes the intraclass compactness and another penalty graph which characterizes the interclass separability. The marginal Fisher criterion is defined as
W∗ = argmin W tr(WTX(D−A)XTW) tr(WTX(Dp −Ap)XTW)
(4)
where D and Dp are diagonal matrices with elements Dii = ∑ j Aij , and D p ij = ∑ j A p ij , respectively. Then we can learn the projection matrix to multiply PCA’s projection and marginal Fisher projection,
WMFA = WPCAW ∗ (5)
3.3 MARGINAL DEEP ARCHITECTURES (MDA)
In order to combine the advantages of MFA and proposed deep architectures, we propose the marginal deep architectures (or MDA). The MDA inherited from the proposed novel framework of deep architectures is shown in Fig. 1. As an input vector x ∈ [0, 1]d, we first map it to higher dimensional space by a random weight matrix Wr1. The representation of first hidden layer is computed as
a1 = s(WTr1x+ b) (6)
where, s(.) is the sigmoid function s(x) = 11+e−x , b is the bias terms, a 1 is the output of first layer. From second layer to (n − 1)-th layer, we use the weight matrices learned from MFA to map layer by layer.
ak = s(WTMFAk−1a k−1 + b) (7)
The last layer is a softmax regression layer and the number of neuron is the number of category. The cost function is defined as,
J(w) = − 1 N ( N∑ i=1 K∑ j=1 I(yi = j) log exp(wTj a n−1 i )∑K l=1 exp(w T l a n−1 i ) ) (8)
where, I(x) is the indicator function, I(x) = 1 if x is true, else I(x) = 0. yi is the label corresponding to xi. Then the probability that xi is classified to j is,
p(yi = j|xi,w) = exp(wTj a n−1 i )∑K
l=1 exp(w T l a n−1 i )
(9)
Taking derivatives, one can show that the gradient is,
∇J(w) = − 1 N N∑ i=1 [xi(I(yi = j)− p(yi = j|xi,w)] (10)
If the n−1 layer’s neurons are more than the last layer, we can continue using MFA to map it. On the contrary, If the n − 1 layer’s neurons are less than last layer, we can randomly initialize the weight matrix between this two layers. Next, in order to improve the MDA, we introduce back propagation, denoising and dropout operation.
3.4 BACK PROPAGATION
In order to adjust the network, we use back propagation (Rumelhart et al., 1986) to compute partial derivative and stochastic gradient descent to update the weight matrixes and the bias terms. For each node i in output layer (n-th layer), we compute an error term as
δni = ∇J(w) (11)
where, J(w) is the cost function computed from Equ.8 and ∇J(w) computed from Equ.10. For each node i in (n− 1)-th to second layer, the error term is computed as,
δki = ( k+1∑ j=1 wkjiδ k+1 j )s ′(zki ) (12)
The back propagation procedure relies on computing the gradient of an objective function with respect to the weights of a multilayer stacked modules. It starting from the output at the top and end to the input at the bottom.
3.5 DENOISING OPERATION
Vincent et al. proposed the denoising autoencoder to improve the robustness of autoencoder (Vincent et al., 2008). It’s very similar with the regularization methods and avoids the “overfitting” problem. The basic idea is to corrupt partial input data by the desired proportion of ν “destruction”. for each input x, a fixed number νd of components are chosen at random, and their value is forced to 0, while the others are left untouched. The initial input x to get a partially destroyed version x̃ by means of a stochastic mapping, x̃ ∼ qD(x̃|x) (13) where, qD(x̃|x) is the unknown distribution. Then, for a hidden representation h,
h = s(WT x̃+ b) (14)
In our MDA, we use this idea to improve the network, please refer to Fig. 1 to find clear sight. For the input layer, the output of first hidden layer is represented as,
a2 = s(WTr1 x̃+ b1) (15)
where, Wr1 is the first layer random weight matrix, b1 is the bias term of first layer. The “denoising” operation is established to a hypothetical additional specific criterion: robustness to partial destruction of the input, which means a good intermediate representation is learned from unknown distribution of its observed input. This operation helps for learning more stable structure and avoids the overfitting problem in most cases.
3.6 DROPOUT
As the same reason with denoising operation, dropout is a trick to prevent overfitting (Hinton et al., 2012b). When a large feedforward neural network is trained on a small training set, dropout performed well on test set. In order to prevent the complex co-adaptations on the training data, the basic idea of dropout is that each hidden node is randomly omitted from the network with a probability of β, so a hidden node can’t rely on other hidden node. In another view, dropout is as a very efficient way of performing model averaging with neural networks. On test set, we train many separate networks and then to apply each of these networks to the test data. Dropout operation can save the train time and then we average the predictions produced by a very large number of different networks. Fig. 1 shows the dropout operation in our MDA.
4 EXPERIMENTS
4.1 DATESET DESCRIPTIONS
We evaluate the performance of MDA on five benchmark data sets. The detail of the data is showed in Tab 1. The USPS 1 data set is a handwritten digits image data set includes 7291 training samples and 2007 test samples from 10 classes with 256 dimensional features. This task is to recognize the digits 0 to 9. The Isolet 2 data set is a collection of audio feature vectors of spoken letters from the English alphabet. It includes 6238 training samples and 1559 test samples from 26 classes with 614 dimensional features. The task is to identify which letter is spoken based on the recorded
1http://www.gaussianprocess.org/gpml/data/ 2http://archive.ics.uci.edu/ml/datasets/ISOLET
(and pre-processed) audio signal. Sensor 3 is a sensorless drive diagnosis data set includes 46816 training samples and 11693 test samples from 11 classes with 48 dimensional features. The features are extracted from electric current drive signals. The task is to classify 11 different classes with different conditions of the drive which has intact and defective components. Covertype 4 contains geological and map-based data from four wilderness areas located in the Roosevelt National Forest of northern Colorado. It includes 15120 training samples and 565892 test samples from 7 classes with 54 dimensional features. The task is to identify forest cover type from cartographic variables. For the IbnSina 5 ancient Arabic document data set, we use 50 pages of the manuscript for training (17543 training samples) and 10 pages for testing (3125 test samples). The data samples belong to 174 classes of subwords and are of dimensionality 200.
In addition, we also use a large scale dataset CIFAR-10 6 to test our MDA on large scale applications. The CIFAR-10 dataset consists of 60000 32 × 32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. We also test our MDA on a specific task which use the CMU motion capture (CMU mocap) data set 7. The CMU mocap data set includes three categories, namely, jumping, running and walking. We choose 49 video sequences from four subjects. For each sequence, the features are generated using Lawrences method 8, with dimensionality 93 (Zhong et al., 2010). By reason of the few samples of CMU, we adopt 10-fold cross-validation in our experiments and use the average error rate and standard deviation to evaluate the performance.
4.2 CLASSIFICATION ON FIVE BENCHMARK DATA SETS
4.2.1 BASELINE METHODS
In order to evaluate the performance of MDA, we compared our MDA with 5 deep learning models include autoencoder (AE) (Hinton & Salakhutdinov, 2006), stacked autoencoders, denoising autoencoders (Vincent et al., 2008), stacked denoising autoencoders (Vincent et al., 2010) and stacked denoising autoencoders with dropout, 2 feature learning models, MFA (Zhong et al., 2013; Yan et al., 2007) and PCA (Jolliffe, 2002), PCA deep architecture base on our uniform framework and the classification accuracy on original space.
4.2.2 EXPERIMENTAL SETTINGS
All of the deep learning methods have the same settings. The size of minibatch was set to 100, the learning rate and momentum were the default value 1 and 0.5, the number of epoch was set to 400, the dropout rate and denoising rate ν were set to 0.1. For the AE and SAE, weight penalty of the L2 norm was set to 10−4. For MFA, the number of nearest neighbors for constructing the intrinsic graph was set to 5, while that for constructing the penalty graph was set to 20. The target spaces of MFA and PCA on different data sets were showed in Tab 1. For the USPS data set, The architecture was set to 256 − 512 − 256 − 128 − 64 − 32. For the Isolet data set ,the architecture was set to 617 − 1324 − 617 − 308. For the Sensor data set, the architecture was set to 48 − 96 − 48 − 24.
3http://archive.ics.uci.edu/ml/datasets/Dataset+for+Sensorless+Drive+Diagnosis# 4http://archive.ics.uci.edu/ml/datasets/Covertype 5http://www.causality.inf.ethz.ch/al data/IBN SINA.html 6http://www.cs.toronto.edu/ kriz/cifar.html 7http://http://mocap.cs.cmu.edu/ 8http://is6.cs.man.ac.uk/∼neill/mocap/
For the Covertype data set, we set the architecture to 54− 216− 108− 54− 27. Finally, for Ibnsina data set, the architecture was set to 200− 400− 200− 100.
4.2.3 CLASSIFICATION RESULTS
The experimental results are shown in Tab. 2. We can see that our MDA achieves the best results on four dataset except the Sensor dataset, but MDA achieves the second best result on Sensor data set and only below the PDA. The PDA achieves the best result on Sensor data set and the second best results on other data sets. These results demonstrate that our uniform deep architectures achieve the good performance in most case. In addition, MDA not only outperform the traditional deep learning models, but also the shallow feature learning models. It shows that our deep architectures based on stacked some feature learning models can learn the better feature than shallow feature learning models.
4.3 EVALUATION
4.3.1 DIFFERENT STRUCTURES FOR MDA
In order to evaluate the desired structures of MDA, we changed the node’s number of the second layer. For USPS data set, we get rid of the second layer and the architecture was 256−128−64−32. Then, we set the number of node of the second layer was as twice as the input layer, the architecture was 256 − 128 − 64 − 32. Next, the number of node was as quadruple as the input layer, the architecture was 256− 1024− 512− 256− 128− 64− 32. Finally, the node’s number is as octuple as the input layer, the architecture was 256 − 2048 − 1024 − 512 − 256 − 128 − 64 − 32. The structures of other data sets are shown in Tab. 3.
The experimental results are shown in Tab. 4. When the the number of nodes of the second layer is as twice as the input layer, MDA achieved the minimum classification error on all data sets except the Covertype data set. When the number of nodes of the second layer is as quadruple as the input
layer, MDA get the worst result on Covertype data set. We can conclude that MDA can work well when the number of nodes of the second layer is as twice or quadruple as the input layer.
4.3.2 DIFFERENT NUMBER OF HIDDEN LAYERS FOR MDA
In order to evaluate how many hidden layers adapt to different datasets, we designed some experiments which have different number of hidden layers. We used 1 ∼ 7 hidden layers on USPS and Isolet datasets and 1 ∼ 5 hidden layers on Covertype, Sensor and Ibnsina datasets. The experimental settings were same as previous experiments.
Tab. 5 shows the classification error on 5 datasets with different hidden layers. All the datasets achieved the best results when hidden layer’s number is 3 except USPS dataset. The USPS dataset achieved the best result when hidden layer’s number is 5. As 1 ∼ 3 hidden layers, with the increase of the number of layers, the classification error is decreasing on all datasets. As small and middle scale applications, we don’t need very deep architectures to handle it. As large scale applications, we can design deeper architectures to achieve better performance.
4.4 CLASSIFICATION ON LARGE SCALE DATASET CIFAR-10
The previous section introduced the advantages of MDA on small and middle scale applications. In order to evaluate the universality of MDA, we chose a relatively large scale dataset CIFAR-10 to test the performance of MDA.
In our experiments, we first transformed the color images to gray images in order to reduce the dimensionality of input. Then we took one sample as a 1024 dimensional vector which is the input of our MDA. So, we can call this data set gray-CIFAR10. The architecture was set to 1024−2048− 1024−512−256−128−64, the minibatch’s size was set to 100, the dropout ratio and denoising ratio were set to 0.1, the number of epoch was set to 400, the learning rate was set to 1, the momentum was set to 0.5. We compared our MDA with previous 6 methods.
Table. 6(a) shows the classification error on gray-CIFAR10, we can see that PDA and MDA achieved the best results in these 7 methods. However, all of the methods on this framework didn’t perform well because we use the gray operation.
4.5 CLASSIFICATION ON CMU MOCAP DATA SET
CMU mocap data set is a very small dataset that only has 49 samples. Traditional deep learning methods didn’t work well in these kind of applications. We test our MDA and PDA and compared them with other 5 deep learning models. The architectures for all deep models (except the PDA) were set to 93− 186− 93− 47− 24. Specially, since the CMU mocap data set only has 49 samples, the PCA method only reduce the dimensionality to 49 at most, so the architecture of PDA was set to
93 − 186 − 24. The denoising ratio and dropout ratio were set to 0.1 on DAE, DAE with dropout, SDAE, SAE, PDA and MDA. The weight penalty on AE was set to 10−4. The learning rate was set to 0.01, the momentum was set to 0.5 and the number of epoch is set to 600. The experiment was test on 10-fold cross validation. The experimental results are shown in Tab. 6(b).
In Tab. 6(b), our PDA and MDA achieved the best results in this dataset and have lower standard deviation than other deep learning models. It demonstrates that our PDA and MDA are more stable than other deep learning models. The traditional autoencoder, SDAE, DAE with dropout achieved the same result in this dataset and better than SAE and DAE.
5 CONCLUSION
In this paper, we proposed a novel deep learning framework that based on stacked some feature learning models to handle small or middle data sets. Then we introduce MFA in this framework, called MDA. The deep learning tricks like backpropagation, denoising and dropout operation are applied on MDA to improve its performance. Extensive experiments on 7 different type data sets demonstrate that MDA performs not only better than shallow feature learning models, but also stateof-the-art deep learning models on small and middle scale applications. The evaluation of MDA show that how to adjust the parameters make the MDA work well. For future work, we plan to try other feature learning models and explore the different structures for this novel deep learning model. In addition, we plan to explore new deep architectures based on this framework to handle the large scale datasets.
The proposed approach consists in a greedy layer wise initialization strategy for a deep MLP model, which is followed by global gradient-descent with dropout for fine-tuning. The initialization strategy uses a first randomly initialized sigmoid layer for dimensionality expansion followed by 2 sigmoid layers whose weights are initialized by Marginal Fisher Analysis (MFA) which learns a linear dimensionality reduction based on a neighborhood graph constructed using class label information (i.e. supervised dimensionality reduction). Output layer is a standard softmax layer.
The approach is thus to be added to a growing list of heuristic layer-wise initialization schemes.
The particular choice of initialization strategy, while reasonable, is not sufficiently well motivated in the paper relative to alternatives, and thus feels rather arbitrary.
The paper lacks clarity in the description of the approach:  MFA is poorly explained with undefined notations (in Eq. 4, what is A? It has not been properly defined); the precise use of alluded denoising in the model is also unclear (is there really training of an additional denoting objective, or just input corruption?).
The question of the (arguably mild) inconsistency of applying a linear dimensionality reduction algorithm, that is trained without any sigmoid, and then passing its learned representation through a sigmoid is not even raised. This, in addition to the fact that sigmoid hidden layers are no longer commonly used (why did you not also consider using RELUs?).
More importantly I suspect methodological problems with the experimental comparisons: the paper mentions using *default* values for learning-rate and momentum, and having (arbitrarily?) fixed epoch to 400 (no early stopping?) and L2 regularization to 1e-4 for some models. 
*All* hyper parameters should always be properly hyper-optimized using a validation set (or cross-validation) including early-stopping, and this separately for each model under comparison (ideally also including layer sizes). This is all the more important since you are considering smallish datasets, so that the various initialization strategies act mainly as different indirect regularization schemes: they thus need to be carefully tuned. This casts serious doubts as to the amount of hyper-parameter tuning (close to none?) that went into training the alternative models used for comparison. 
The Marginal Fisher Analysis dimensionality reduction initialization strategy may well offer advantages, but as it currently stands this paper doesn’t yet make a sufficiently convincing case for it, nor provide useful insights into the nature of the expected advantages.
I would also suggest, for image inputs such as CIFAR10, to use the qualitative tool of showing the filters (back projected to input space) learned by the different initialization schemes under consideration, as this could help visually gain insight as to what sets methods apart.