Classifying question sentences into their corresponding categories is an important task with wide applications, for example in many websites’ FAQ sections. However, traditional question classification techniques do not fully utilize the wellprepared answer data which has great potential for improving question representation and could lead to better classification performance. In order to encode answer information into question representation, we first introduce novel group sparse autoencoders which could utilize the group information in the answer set to refine question representation. We then propose a new group sparse convolutional neural network which could naturally learn the question representation with respect to their corresponding answers by implanting the group sparse autoencoders into the traditional convolutional neural network. The proposed model show significant improvements over strong baselines on four datasets.
1 INTRODUCTION
Question classification has applications in question answering (QA), dialog systems, etc., and has been increasingly popular in recent years. Most existing approaches to this problem simply use existing sentence modeling frameworks and treat questions as general sentences, without any special treatment. For example, several recent efforts employ Convolutional Neural Networks (CNNs) to achieve remarkably strong performance in the TREC question classification task as well as other sentence classification tasks such as sentiment analysis (Kim, 2014; Kalchbrenner et al., 2014; Ma et al., 2015).
We argue, however, that the general sentence modeling frameworks neglect several unique properties in question classification not found in other sentence classification tasks (such as sentimental classification or sarcasm detection), which we detail below:
• The categories for most sentence classification tasks are flat and coarse (notable exceptions such as the Reuters Corpus RCV1 (Lewis et al., 2004) notwithstanding), and in many cases, even binary (i.e. sarcasm detection). However, question sentences commonly belong to multiple categories, and these categories often have a hierarchical (tree or DAG) structure such as those from the New York State DMV FAQ section 1 in Fig. 1.
• Question sentences from different categories often share similar information or language patterns. This phenomenon becomes more obvious when categories are hierarchical. Fig. 2 shows one example of questions sharing similar information from different categories. This cross-category shared patterns are not only shown in questions but can also be found in answers corresponding to these questions.
• Another unique characteristic for question classification is the well prepared answer set with detailed descriptions or instructions for each corresponding question category. These answer sets generally cover a broader range of vocabulary (than the questions themselves) and carry more distinctive semantic meanings for each class. We believe there is great
1 http://nysdmv.custhelp.com/app/home
potential to enhance the representation of questions with extra information from corresponding answer sets.
To exploit the hierarchical and overlapping structures in question categories and extra information from answer sets, we consider dictionary learning (Aharon et al., 2005; Roth & Black, 2005; Lee et al., 2007; Candè & Wakin, 2008; Kreutz-Delgado et al., 2003; Rubinstein et al., 2010) which is one common approach for representing samples from a vast, correlated groups with external information. This learning procedure first builds a dictionary with a series of grouped bases. These bases can be initialized randomly or from external data (from the answer set in our case) and optimized during training through Sparse Group Lasso (SGL) (Simon et al., 2013). There are many promising improvements which have been achieved recently by this grouped-dictionary learning-based methods (Zhao et al., 2016; Rao et al., 2016). We also showcase some preliminary experiments in Section 6 for question classification with SGL, and the performance is indeed extraordinary compared with baselines but still lose to the CNNs-based method. Considering the unique advantages from the SGL-based model and the CNNs-based model, we believe that performance of question classification will have another boost if we could put SGL-based and CNNs-based model within the same end-to-end framework. This requires us to design a new neural-based model which behaves similarly with SGL.
Based on the above observations, we first propose a novel Group Sparse Autoencoders (GSA). The objective of GSA and SGL are very similar. The encoding matrix of GSA (like the dictionary in SGL) is grouped into different categories. The bases in different groups can be either initialized randomly or by the sentences in corresponding answer categories. Each question sentence will be reconstructed by some bases within a few groups. To the best of our knowledge, GSA is the first full neural network based model with group sparse constraints. GSA has can be either linear or nonlinear encoding or decoding while SGL is restrained to be linear. In order to incorporate both advantages from GSA and CNNs, we then propose a new Group Sparse Convolutional Neural Networks (GSCNNs) by implanting the GSA into CNNs between the convolutional layer and the classification layer. GSCNNs are jointly trained end-to-end neural-based framework for getting question representations with group sparse constraint from both answer and question sets. Experiments show significant improvements over strong baselines on four datasets.
2 PRELIMINARIES: SPARSE AUTOENCODERS
We first review the basic autoencoders and sparse autoencoders to establish the mathematical notations. Then we propose our new autoencoder with group sparse constraints in later section.
2.1 BASIC AUTOENCODERS
As introduced in (Bengio et al., 2007), autoencoder is an unsupervised neural network which could learn the hidden representations of input samples. An autoencoder takes an input instance z ∈ Rd, and then maps it into a hidden space in the form of h ∈ Rs through a deterministic mapping function h = Φθ(z) = Φ(Wz + b), where θ = {W, b}. W is a d × s projection matrix and b is the bias term. The projection function can be linear or non-linear function such as sigmoid. This projection process often can be recognized as encoding process. The encoded hidden representation is then mapped back to the original input space to reconstruct a vector ẑ ∈ Rd with function ẑ = Φθ′(h) = Φ(W ′h + c) with θ′ = {W ′, c}. The reverse projection matrix W ′ may optionally be constrained by W ′ = WT . This reverse operation can be recognized as a decoding process which tries to reconstruct a new vector z such that the difference between ẑ and z are as small as possible by minimizing the average reconstruction error:
J(W, b, c) = argmin W,b,c
1
n n∑ i=1 L(z(i), ẑ(i)) = argmin W,b,c 1 n n∑ i=1 L ( z(i),ΦWT ,c(ΦW,b(z (i))) )
(1)
where L is a loss function such as minimum square error L(z, ẑ) = ‖z− ẑ‖2. Depending on the applications, this loss function also can be defined in form of computing the reconstruction cross-entropy between z and ẑ:
LC(z, ẑ) = − d∑
k=1
(zk log ẑk + (1− zk) log(1− ẑk))
When the dimensionality of the hidden space s is smaller than the dimensionality of the input space d. The network is forced to learn a compressed representation of the input. If there is structure or feature correlation in the data, the linear autoencoders often ends up learning a low-dimensional representation like PCA. Most of the time, autoencoders learns a compressed representation when the number of hidden units s being small. However, when the number of hidden units becomes larger than the dimensionality of input space, there are still some interesting structure that can be discovered by imposing other constraints on the network. The following discussed sparse constraints is one of them.
2.2 SPARSE AUTOENCODERS
Sparse autoencoders (Ng, 2011; Makhzani & Frey, 2014) shows interesting results of getting visualization of the hidden layers. Recall that hij represents the activations of j
th hidden unit for a given specific input zi. Then the average activation of hidden unit j (average over the training batch) can be defined as:
ρ̂j = 1
m m∑ i=1 hij (2)
wherem is the number of samples in training batch. The goal of sparse autoencoders is to enforce the constraint:
ρ̂j = ρ (3)
where ρ is the sparsity parameter which controls how sparse you want the hidden representation to be. Typically ρ is set to be a small value close to zero. In order to satisfy this constraint, the activations of hidden layer must mostly be close to 0.
In order to achieve the above objective, there will be an extra penalty term in our optimization function which tries to reconstruct the original input with as few hidden layer activations as possible. The most commonly used penalty term (Ng, 2011) is as follows:
s∑ j=1 KL(ρ||ρ̂j) = s∑ j=1 ρ log ρ ρ̂j + (1− ρ) log 1− ρ 1− ρ̂j (4)
where s is the number of units in hidden layer, and j is the index of the hidden unit. This penalty term is based on KL divergence which measures the difference between two different distributions.
Then our new objective of the sparse autoencoders is defined as follows:
Jsparse(W, b, c) = J(W, b, c) + α s∑ j=1 KL(ρ||ρ̂j) (5)
where J(W, b, c) is defined in Eq. 1, and α controls the weights of the sparsity penalty term. Note that the term ρ̂j is implicitly controlled by W , b and c. This is one of the difference between sparse autoencoders and sparse coding which will be discussed in details in Section 6.
3 GROUP SPARSE AUTOENCODERS
As described above, sparse autoencoder has similar objective with sparse coding which tries to find sparse representations for input samples. Inspired by the motivations from group sparse lasso (Yuan & Lin, 2006) and sparse group lasso (Simon et al., 2013), we propose a novel Group Sparse Autoencoders (GSA)in this paper.
Different from sparse autoencoders, in our GSA, the weight matrix is categorized into different groups. For a given input, GSA reconstructs the input signal with the activations from only a few groups. Similar to the average activation defined in Eq. 2 for sparse autoencoders, in GSA, we define each grouped average activation for the hidden layer as follows:
η̂p = 1
mg m∑ i=1 g∑ l=1 ||hip,l||2 (6)
where g represents the number of samples in each group, and m represents the number of samples in training batch. η̂j first sums up all the activations within pth group, then computes the average pth group respond across different samples’ hidden activations.
Similar with Eq.4, we also use KL divergence to measure the difference between estimated intra-group activation and goal group sparsity as follows:
G∑ p=1 KL(η||η̂p) = η log η η̂p + (1− η) log 1− η 1− η̂p (7)
whereG is the number of groups. When we only need inter-group constraints, the loss function of autoencoders can be defined as follows:
Jgs(W, b, c) = J(W, b, c) + β g∑ l=1 KL(η||η̂p) (8)
In some certain cases, inter- and intra- group sparsity are preferred and the same time. Then objective can be defined as follows:
Jgs(W, b, c) = J(W, b, c) + α s∑ j=1 KL(ρ||ρ̂j)
+ β G∑ p=1 KL(η||η̂p) (9)
Inter-group sparse autoencoders defined in Eq. 8 has similar functionality with group sparse lasso in (Yuan & Lin, 2006). Inter- and intra- group sparse autoencoders which defined in Eq. 9 behaves similarly to sparse group lasso in (Simon et al., 2013). Different from the sparse coding approach, the encoding and decoding process could be nonlinear while sparse coding is always linear.
Similar to sparse coding approach, the projection matrix in GSA works like a dictionary which includes all the necessary bases for reconstructing the input signal with the activations in the hidden layer. Different initialization methods for projection matrix are described in Section 5.
3.1 VISUALIZATION FOR GROUP SPARSE AUTOENCODERS
In order to have a better understanding of how the GSA behaves, We use MNIST dataset for visualizing the internal parameters of GSA. We visualize the projection matrix in Fig. 3 and the corresponding hidden activation in Fig. 4.
In our experiments, we use 10, 000 samples for training. We set the size of hidden layer as 500 with 10 different groups for GSA. We set the intra-group sparsity ρ equal to 0.3 and inter-group sparsity η equal to 0.2. α and β are equal to 1. On the other hand, we also train the same 10, 000 examples on basic autoencoders with random noise added to the input signal (denoising autoencoders (Vincent et al., 2008)) for better hidden information extraction. We add the same 30% random noise into both models. Note that the group size of this experiments does not have to be set to 10. Since this is the image dataset with digit numbers, we may use fewer groups to train GSA.
In Fig. 3(b), we could find similar patterns within each group. For example, the 8th group in Fig. 3(b) has different forms of digit 0, and 9th group includes different forms of digit 7. However, it is difficult to tell any meaningful patterns from the projection matrix of basic autoencoders in Fig. 3(c).
Fig. 4 shows the hidden activations respect to the input image in Fig. 3(a). From the results, we can tell that most of the activations of hidden layer are in group 1, 2, 6 and 8. And the 8th group has the most significant activations. When we refer this activations to the projection matrix visualization in Fig. 3(b). These results are reasonable since the 8th row has the most similar patterns of digit 0.
GSA could be directly applied to small image data (i.e. MINIST dataset) for pre-training. However, in the tasks which prefer dense, semantic representation (i.e. sentence classification), we still need CNNs to learn the
sentence representation automatically. In this scenario, in order to incorporate both advantages from GSA and CNNs, we propose Group Sparse Convolutional Neural Networks in the following section.
4 GROUP SPARSE CONVOLUTIONAL NEURAL NETS
Convolutional neural networks (CNNs) were first proposed by (LeCun et al., 1995) in computer vision. For a given image, CNNs apply convolution kernels on a series of continuous areas on images. This concept was first adapted to NLP by (Collobert et al., 2011). Recently, many CNNs-based techniques achieve great successes in sentence modeling and classification (Kim, 2014; Kalchbrenner et al., 2014; Ma et al., 2015). For simplicity, we use the sequential CNNs (Kim, 2014) as our baseline.
Following sequential CNNs, one dimensional convolution operates the convolution kernel in sequential order in Eq. 10, where xi ∈ Re represents the e dimensional word representation for the i-th word in the sentence, and ⊕ is the concatenation operator. Therefore xi,j refers to concatenated word vector from the i-th word to the (i+ j)-th word in sentence:
xi,j = xi ⊕ xi+1 ⊕ · · · ⊕ xi+j (10)
A convolution operates a filter w ∈ Rn×e to a window of n words xi,i+n with bias term b′ described in Eq. 11 to produce a new feature.
ai = σ(w · xi,i+n + b′) (11)
where σ is a non-linear activation function such as rectified linear unit (ReLu) or sigmoid function. The filter w is applied to each word in the sentence, generating the feature map a ∈ RL:
a = [a1, a2, · · · , aL] (12)
where L is the length of the sentence.
The convolution described in Eq. 11 can be regarded as feature detection: more similar patterns will return higher activation. In sequential CNNs, max-over-time pooling (Collobert et al., 2011; Kim, 2014) operates over the feature map to get the maximum activation â = max{a} representing the entire feature map. The idea is to detect the strongest activation over time. This pooling strategy also naturally deals with sentence length variations.
In order to capture different aspects of patterns, CNNs usually randomly initialize a set of filters with different sizes and values. Each filter will generate a feature as described above. To take all the features generated by N different filters into count, we use z = [â1, · · · , âN ] as the final representation.
In conventional CNNs, z will be directly fed into classifiers after the sentence representation is obtained, e.g. fully connected neural networks in (Kim, 2014). There is no easy way for CNNs to explore the possible hidden representations with interesting underlaying structures.
In order to obtains the hidden representations for each sentence representation, we proposed a Group Sparse Convolutional Neural Networks (GSCNNs) by placing one extra layer between convolutional layer and classification layer. This extra layer is trying to mimic the functionality of GSA that we introduced in Section 2.
Our proposed framework is shown in Fig. 5. The convolutional layer show in Fig. 5 follows the traditional convolution process which is described previously. After the convolutional layer, we get z which is the feature map for each sentence. The feature maps z is treated as the feature representation for each sentence. In stead of directly feeding z into a fully connected neural network for classification, we enforce the group sparse constraint on z like the group sparse constraint we have on h in Eq. 9. Then, we use the hidden representation h in Eq. 9 as new sentence representation. The last step is feeding the hidden representation h into fully connected neural network for classification. The parameters W , b, and c in Eq. 9 will also be fine tunned during the last step.
In order to improve the robustness of the hidden representation and prevent it from simply learning the identity, we follow the idea of decisioning autoencoders (Vincent et al., 2008) to add random noise (10% in our experiments) into z. The training process of our model is similar to the training process in stack autoencoders (Bengio et al., 2007).
In order to prevent the co-adaptation of the hidden unites, we employ random dropout on penultimate layer (Hinton et al., 2014). We set the drop out rate as 0.5 and learning rate as 0.95 by default. In our experiments, training is done through stochastic gradient descent over shuffled mini-batches with the Adadelta update rule (Zeiler, 2012). All the settings of the CNNs are the same as the settings in (Kim, 2014).
5 EXPERIMENTS
Since there has been little effort to use answer sets in question classification, we did not find any well-fitted datasets which are publicly available. We collected two datasets and use other two well-known datasets in our experiments. The statistics of these datasets is summarized in Table 1. The descriptions of each dataset are as follows:
• TREC The TREC dataset2 is a factoid question classification dataset. The task is to classify each question into one of the 6 different question types (Li & Roth, 2002). The reason we include this factoid questions dataset is to show the effectiveness of the proposed method in an frequently used dataset even there is no categorized answer sets available.
• Insurance This is a private dataset which we collected from a car insurance company’s website. Each question is classified into the 319 possible classes with corresponding answer data. All questions which belongs to the same category share the same answers. All answers are generated manually. Most questions have multiple assigned labels.
• DMV dataset We collected this dataset from New York State DMV’s FAQ website. We will make this data publicly available in the future.
• Yahoo Ans The Yahoo! Answers dataset (Fleming et al., 2012; Shah & Pomerantz, 2010) is a publicly available dataset.3 There are more than 4 million questions with answers. For simplicity reasons, we only randomly sample 8,871 questions from the complete dataset. There are 27 top level categories across different domains. To make our task more realistic and challenging, we test the proposed model with respect to the subcategories and there are 678 classes.
2http://cogcomp.cs.illinois.edu/Data/QA/QC/ 3http://webscope.sandbox.yahoo.com/catalog.php?datatype=l
We only compare our model’s performance with CNNs for two following reasons: we consider our “group sparse” as a modification to the general CNNs for grouped feature selection. This idea is “orthogonal” to any other CNNs-based models and can be easily applied to them; another reason is, as discussed in Sec. 1, we did not find any other model which can be used for comparison in soloving question classification task with answer sets.
The datasets we use in the experiments require the label information for both questions and answers. Besides that, similar with websites’ FAQ section, all the questions which belong to the same category share the same answer sets. Among the above the four datasets, only the Insurance and DMV datasets are well-fitted for our model. The questions which fall into the same category have different answers in Yahoo dataset.
Different ways of initializing the projection matrix in Eq. 9 can be summarized as the followings:
• Random Initialization: when there is no answer corpus available, we first random initialize N vectors (usually N s) to represent the representation from answer set. Then we cluster these N vectors intoG categories with g centroids for each category. These centroids from different categories will be the initialized bases for projection matrixW . This projection matrix will be optimized during training.
• Initialization from Questions: instead of using random initialized vectors, we could also use question sentences for initializing the projection matrix when answer set is not available. We need to pre-train the sentence with CNNs to get the sentence representation. We then select top G largest categories in terms of number of question sentences. Then we get g centroids from each category by k-means. We concatenate these G × g vectors group after group to form the projection matrix. We need to pre-train the sentence with CNNs to get the sentence representation.
• Initialization from Answers: This is the most ideal case. We follow the same procedure from above. The only difference is that we then treat the answer sentence as question sentence to pre-train the CNNs to get answer sentence representation.
Note that projection matrix will be updated during training for better classification performance.
In the cases of single-label classification tasks (TREC and Yahoo dataset), we set the last layer as softmax-layer which tries to get one unique peaky choice across all other labels. But in the cases for multi-label classification (Insurance and DMV dataset), we replace the softmax-layer in CNNs with sigmoid-layer since sigmoid layer predicts each category independently while softmax function has an exclusive property which allows cross influence between categories.
All the experiments results are summarized in Table 2. TREC dataset is factoid question type classification. We include this experiments to show our performance on a frequently used dataset. Proposed method improves marginally over baseline because the sentences are too short in TREC dataset. For Insurance and DMV dataset, the improvement is significant.
In the experiments with Yahoo dataset, the improvement is not as signification as Insurance and DMV. One reason for this is the questions in Yahoo dataset are usually too short, sometime only have 2 to 3 words. When the sentences become shorter, the group information become harder to encode. Another reason is that the questions in Yahoo dataset are always single labeled, and can not fully utilize the benefits of group sparse properties. Yahoo-top shows the results of top categories classification results. We map the subcategories back to the top categories and get the results in Table 2.
Besides the conventional classification tasks, we also test our proposed model on unseen-label experiments. In this experiments, there are a few sub-category labels that are not included in training process. However, we still hope that our model could correctly classify these unseen sub-category label into correct parent category based on the model’s sub-category estimation. In the testing set of Yahoo dataset, we randomly add 100 questions
whose labels are unseen in training set. The classification results of Yahoo-unseen in Table 2 are obtained by mapping the subcategory classification results to top level category and check whether the true label’s top category match with predicted label’s parent category. The improvements are remarkable due to the group information encoding.
6 DISCUSSION
The idea of reforming signal to a sparse representation is first introduced in the domain of compressed sensing (Candè & Wakin, 2008) which achieves great success in signal compression, visualization and classification task. Especially when dictionary is well trained, the performance usually improves significantly, as shown in (Wang et al., 2010; Yang et al., 2009) for image classification tasks. In Table 3, we test the influence of Sparse Group Lasso (SGL) (Simon et al., 2013) with two baseline methods, k-Nearest Neighbor (k-NN) and SVM on the Insurance dataset. We use TF-IDF as feature representation for each question and answer sentence. We first select all the answer sentences from top 20 largest category and then find 10 centroids for each of these categories by k-Means. Then we have a dictionary with 200 centroids with 20 groups. We notice there is a great improvement of performance after we preprocess the original sentence representations with SGL before we use SVM. We further test the performance of CNNs on the same dataset, and CNNs outperforms SVM and k-NN even with SGL because of the well trained sentence representation through CNNs. However, for vanilla CNNs, it is not straightforward to embed SGL into the network and still get good representation for sentences since SGL will break the training error in backpropagation.
However, GSA is fully neural network based framework. Our proposed GSA has similar functionalities to SGL (Yuan & Lin, 2006; Simon et al., 2013), as it is shown in Fig. 3 and Fig. 4, but in different approach. Compared with sparse coding approaches which have intense optimizations on both dictionary and coding, GSA’s optimization is based on simple backpropagation. GSA also can be easily placed into any neural network for joint training. Another advantage of GSA over sparse coding is that the projection function Φ in GSA can be linear or non-linear, while sparse coding always learns linear codings.
7 CONCLUSIONS AND FUTURE WORK
In this paper, we first present a novel GSA framework which functions as dictionary learning and sparse coding models with inter- and intra- group sparse constraints. We also prove GSA’s learning ability by visualizing the projection matrix and activations. We further propose a group sparse convolutional neural networks by embedding GSA into CNNs. We show that CNNs can benefit from GSA by learning more meaningful representation from dictionary.
The paper proposes the group sparse autoencoder that enforces sparsity of the hidden representation group-wise, where the group is formed based on labels (i.e., supervision). The p-th group hidden representation is used for reconstruction with group sparsity penalty, allowing learning more discriminative, class-specific patterns in the dataset. The paper also propose to combine both group-level and individual level sparsity as in Equation (9). 
Clarity of the paper is a bit low. 
- Do you use only p-th group's activation for reconstruction? If it is true, then for Equation (9) do you use all individual hidden representation for reconstruction or still using the subset of representation corresponding to that class only? 
- In Equation (7), RHS misses the summation over p, and wondering it is a simple typo.
- Is the algorithm end-to-end trainable? It seems to me that the group sparse CNN is no more than the GSA whose input data is the feature extracted from sequential CNNs (or any other pretrained CNNs).
Other comments are as follows:
- Furthermore the group sparse autoencoder is (semi-) supervised method since it uses label information to form a group, whereas the standard sparse autoencoder is fully unsupervised. That being said, it is not surprising that group sparse autoencoder learns more class-specific pattern whereas sparse autoencoder doesn't. I think the fair comparison should be to autoencoders that combines classification for their objective function.
- Although authors claim that GSA learns more group-relevant features, Figure 3 (b) is not convincing enough to support this claim. For example, the first row contains many filters that doesn't look like 1 (e.g., very last column looks like 3).
- Other than visual inspection, do you observe improvement in classification using proposed algorithm on MNIST experiments?
- The comparison to the baseline model is missing. I believe the baseline model shouldn't be the sequential CNN, but the sequential CNN + sparse autoencoder. In addition, more control experiment is required that compares between the Equation (7)-(9), with different values of \alpha and \beta.
Missing reference:
Shang et al., Discriminative Training of Structured Dictionaries via Block Orthogonal Matching Pursuit, SDM 2016 - they consider block orthgonal matching pursuit for dictionary learning whose blocks (i.e., projection matrices) are constructed based on the class labels for discirminative training.