Pooling is an important technique for learning text representations in many neural NLP models. In conventional pooling methods such as average, max and attentive pooling, text representations are weighted summations of the L 1 or L  norm of input features. However, their pooling norms are always fixed and may not be optimal for learning accurate text representations in different tasks. In addition, in many popular pooling methods such as max and attentive pooling some features may be over-emphasized, while other useful ones are not fully exploited. In this paper, we propose an A ttentive P ooling with L earnable N orms (APLN) approach for text representation. Different from existing pooling methods that use a fixed pooling norm, we propose to learn the norm in an end-to-end manner to automatically find the optimal ones for text representation in different tasks. In addition, we propose two methods to ensure the numerical stability of the model training. The first one is scale limiting, which re-scales the input to ensure non-negativity and alleviate the risk of exponential explosion. The second one is re-formulation, which decomposes the exponent operation to avoid computing the real-valued powers of the input and further accelerate the pooling operation. Experimental results on four benchmark datasets show that our approach can effectively improve the performance of attentive pooling. In recent years, neural network based methods are widely used in the natural language processing (NLP) field to learn text representations (Yang et al., 2016; Peters et al., 2018). In these methods, pooling is a core technique to build the text representation vector from a collection of input feature vectors by summarizing their information (Lai et al., 2015). Thus, an effective pooling method Sentiment Classification Average Pooling The movie is good, but not to my taste Max Pooling The movie is good, but not to my taste Attentive Pooling The movie is good, but not to my taste News Topic Classification Average Pooling Fire on Queensland Island Takes Heavy Toll on Wildlife Max Pooling Fire on Queensland Island Takes Heavy Toll on Wildlife Attentive Pooling Fire on Queensland Island Takes Heavy Toll on Wildlife Figure 1: The pooling weights of several different pooling methods on the representations produced by an LSTM network. that can select salient features accurately will facilitate many NLP methods (Ma et al., 2017). Among existing pooling methods, average pooling is a representative one which takes the average of the L 1 norm of input features (Tang et al., 2014, 2015a,b). However, average pooling equally regards the input representation vector at each position and ignores their different informativeness for learning text representation, which may not be optimal (Johnson and Zhang, 2015). Thus, other pooling methods such as max pooling (Col-lobert et al., 2011; Kim, 2014) and attentive pooling (Yang et al., 2016; Zhou et al., 2016; Cui et al., 2017; Devlin et al., 2019; Wu et al., 2019b) are widely used in neural NLP models. For example, Kim (2014) proposed to apply max pooling to the contextual word representations learned by CNN networks to build the representations of the entire sentence. Yang et al. (2016) proposed to use attentive pooling at both word and sentence levels to learn informative sentence and document representations by selecting important words and sentences. However, these pooling methods use fixed average norms, i.e., L 1 norm for average and attentive pooling and L  norm for max pooling, to build text representations, which may not be optimal when handling different tasks. Our work is motivated by the following observations. First, different contexts usually have different informativeness for learning text representations. For example, in Fig. 1 1 , the word but is very important for inferring the sentiment polarity of this sentence, while The is uninformative. Thus, modeling the different informativeness of contexts and attending to them differently may help learn more informative text representations. Second, different tasks and even different datasets have different characteristics. For example, in Fig. 1, sentiment and negation words may be the key clues for inferring the sentiment polarity of the first sentence, while the global contexts may be useful for understanding the topic of the second sentence. Thus, using a fixed pooling norm for universal text representation learning is probably not optimal. Third, in popular pooling methods such as max pooling and attentive pooling, some contexts may be over-emphasized, and other useful contextual information is not fully-respected. For example, as shown in Fig. 1, the sentiment word good is highlighted, but other useful clues such as but and not do not gain sufficient attentions, which may not be optimal for learning accurate text representations. Thus, a dynamically learnable degree of hard or soft for pooling may benefit text representation learning. In this paper, we propose an Attentive Pooling with Learnable Norms (APLN) approach to enhance the learning of text representations 2 . Instead of manually setting a fixed pooling norm, we propose to automatically learn it in a unified framework, which can find the optimal values to learn text representations for different tasks in an end-to-end manner. In addition, since the learning of pooling norm may be numerically unstable in some cases due to the exponent operation, we propose two methods to improve its computational stability. The first one is limiting the scale of input features, which aims to ensure their non-negativity and avoid exponential explosion. The second one is a re-formulation method, which aims to avoid computing the real-valued power of input features by decomposing the exponent operation into three safe and fast atomic operations. We conducted experiments on four benchmark datasets, and the results show that our approach can effectively improve the learning of text representation. Neural networks are widely used to learn text representations from contexts (Peng et al., 2018). Pooling is usually an essential step in these methods to build contextual representations by summarizing the information of input features (LeCun et al., 2015). The simplest pooling method is average pooling, which is used in many approaches to construct text representations (Tang et al., 2014, 2015a,b). For example, Tang et al. (2015a) proposed to apply average pooling to the output of CNN filters to capture global contexts in a sentence. In addition, they also proposed to average the sentence representations learned by parallel CNN networks with different window sizes. In their another work (Tang et al., 2015b), they proposed to apply average pooling to the sequence of sentence representations to build the representations of an entire document. Although average pooling is computationally efficient, it cannot distinguish important contexts from unimportant ones, which may not be optimal for learning accurate text representations. There are also other popular pooling methods that can select salient features to learn more informative text representations, such as max pooling (Kim, 2014; Zhang et al., 2015) and attentive pooling (Yang et al., 2016), which are employed by many neural NLP methods (Collobert et al., 2011; Kim, 2014; Huang et al., 2012; Yang et al., 2016; Chen et al., 2016; Zhou et al., 2016; Du et al., 2017; Li et al., 2018; Wu et al., 2019a; Tao et al., 2019; Devlin et al., 2019; Wu et al., 2019b). For example, Collobert et al. (2011) proposed to learn representations of contexts within each window using feed forward neural networks, and used max pooling to build final text representations. Kim (2014) proposed to apply max pooling over time to the contextual word representations learned by multiple CNN filters. Huang et al. (2012) proposed to build representations of the entire document using the summation of word representations weighted by their TF-IDF scores. Yang et al. (2016) proposed a hierarchical attention network to first learn sentence representations from words and then learn document representations from sentences. They proposed to apply attentive pooling at both word and sentence levels to select informative words and sentences for more informative representation learning. Wu et al. (2019b) proposed a hierarchical user and  1  2   1    1  2   Max-over-time ...   1  2   ...    1  2   ...   1  2   ...       ...  1/      1  2        ... 1  2   1    1  2   Max-over-time ...   1  2   ...    1  2   ...   1  2   ...       ...  1/      1  2        ... item representation model with three-tier attention, which applies attentive pooling to simultaneously select important words, sentences and reviews. However, the pooling norms of max and attentive pooling are always fixed, which may not be optimal for universal text representation learning since the characteristics of different tasks may be different. In addition, both pooling methods may over-emphasize the most salient features, and other useful contextual information is not fully exploited, which may also be sub-optimal. There are a few methods to adapt the pooling norms in different tasks. For example, Gulcehre et al. (2014) explored the influence of selecting different pooling norms on the performance of different image classification tasks. However, the norms in their method are manually tuned, which are usually very time-consuming and may not be optimal. Different from all aforementioned methods, our approach can automatically optimize pooling norms in an end-to-end manner, and can effectively select important contexts to learn informative text representations. Extensive experiments on four datasets with different characteristics validate the effectiveness of our approach. In this section, we will first present a brief introduction to several popular pooling methods, i.e., average, max and attentive pooling. To make it easier to understand, we present an intuitive comparison of the mechanisms of these different pooling methods in Fig. 2. Average Pooling. Average pooling is used to build contextual representations by taking the arithmetic mean of input features, as shown in Fig. 2(a). It uses the L 1 norm of the input. Denote the input sequence of hidden representations as [ h 1 , h 2 , ..., h N ] , where N is the sequence length. Max Pooling. Max pooling aims to build contextual representations by selecting the most salient features via max-over-time operations, as shown in Fig. 2(b). It utilizes the L  norm at the time dimension of input features. Denote r j as the j -th value in the vector r , which is computed as: r j = max( h j 1 , h j 2 , ..., h jN ) , (2) where h ji represents the j -th value in the feature vector h i . Attentive Pooling. As shown in Fig. 2(c), attentive pooling usually builds contextual representations by selecting important input features, which can also be regarded as a kind of L 1 norm average. It computes an attention weight  i for the input at each position to indicate its informativeness, which is formulated as follows:  i = exp[ q T f ( h i )] (cid:80) Nj =1 exp[ q T f ( h j )] , (3) where f (  ) is a non-linear function, q is the attention query vector. Following Yang et al. (2016), we apply the tanh operation to the linear transformation of h i to form the function f (  ) . The final contextual representation r is the summation of input representation vectors weighted by their attention weight as follows: r = N (cid:88) i =1  i h i . In this section, we will introduce the details of our Attentive Pooling with Learnable Norms (APLN) approach. In the aforementioned pooling methods, the pooling norm is always fixed (i.e., L 1 or L  ). However, the characteristics of different NLP tasks and even different datasets should have some differences, and it may not be optimal to use a fixed pooling norm for universal text representation learning. In addition, tuning the pooling norm manually is usually very time-consuming, and it may also be sub-optimal. Thus, it is an intuitive idea to automatically learn the pooling norm in an end-to-end manner to alleviate the efforts on hy-perparameter searching and learn more informative text representations. The architecture of our APLN approach is shown in Fig. 3. We will introduce its details as follows. Since different contexts usually have different importance, modeling their informativeness may help learn more informative text representations. Thus, similar to the vanilla attentive pooling, in our APLN approach, we also compute an attention score for the input at each position. However, instead of using the simple weighted summation to build the contextual representation r , we propose to compute the L p norm 3 average of the input feature vectors weighted their attention weights, which is formulated as follows: r = [ 1 (cid:80) Ni =1  pi N (cid:88) i =1 (  i h i ) p ] 1 p , (5) 3 It should be noticed that when p < 1 , this definition is not a norm since it does not obey the triangle inequality. But we still call it norm for consistency. where p is a learnable parameter. In this way, our model will automatically find appropriate values of pooling norms for learning text representations in different tasks. To show the influence of p on the inputs of the APLN module, we vary the value of p and illustrate the shape of the function y = x p in Fig. 4. According to Fig. 4, we can see when p is larger, the attention of APLN is sharper and sparser since small values of  i h i will be suppressed, which indicates the attentive pooling is harder. In contrast, if p is smaller, the attentions are more distributed, which indicates the attentive pooling is softer. Thus, in this manner, our APLN model can automatically explore how hard/soft the attention should be when constructing text representations, which may help recognize important contexts and avoid the problem of over-emphasizing some features and not fully respecting other useful ones, both of which are important for learning accurate text representations. Unfortunately, in most cases the training of APLN is unstable if we directly use it for pooling. Thus, we propose two methods to ensure the numerical stability of the model training. The first one is scale limiting , which is used to limit the range of the elements of  i h i . The second one is Re-formulation , which is used to avoid the direct computation of the real-valued powers of the input features and accelerate the pooling operation. We will introduce the two methods as follows. According to Eq. (5), to ensure the values of r are real, the elements of  i h i must be non-negative. Thus, we apply a ReLU function to  i h i to keep  i h i  0 . However, there are still some risks if there exist elements with  i h ji > 1 , since the gradients may explode when p > 1 due to the ampli-fication of the exponent, which are also observed in our practice. To solve this problem, we propose to clip the values of  i h i as follows: 0   i h ji  1 . In this way, the input features is re-scaled to a safe range. We also explored other kinds of re-scaling methods such as normalization, but we find there are no significant differences in the model performance. Thus, we simply use the clipping operation for its efficiency. However, there are still some problems in our approach. We find the training of our approach is not numerically stable (e.g., NAN problem) when implemented by several popular deep learning frameworks such as Tensorflow. In addition, computing the real-value powers of input features is quite time-consuming. Thus, we propose a reformulation strategy by converting the exponent computation in Eq. (5). For instance, the exponent x p is re-formulated as follows: x p = e log( x p ) = e p log( x )  e p log( x + (cid:15) ) , (7) where (cid:15) = 10  7 is a protection value. In this way, the computation of the power of x is divided into three atomic operations, i.e., logarithm, multiplication and exponent, all of them are fast 4 and numerically stable in our approach. Thus, using the re-formulation strategy can enhance the numerical stability and accelerate the pooling operation. Our experiments are widely conducted on four benchmark datasets with different characteristics. The first one is AG's News 5 , which is a news topic classification dataset. Following (Zhang et al., 2015), we only use the title and description fields in this dataset. The second one is IMDB 6 (Diao et al., 2014), which is a dataset with movie reviews and ratings. The third one is Amazon Electronics 4 In experiments on a machine with a GTX1080ti GPU, the computation of x p is accelerated by more than 10 times. (denoted as Amazon ) (He and McAuley, 2016), which contains reviews on electronics. The fourth one is Yelp 2015 (denoted as Yelp ), which is a restaurant review dataset. The latter three datasets are all for sentiment classification. Since the original Amazon and Yelp datasets are too large, we sampled 50,000 reviews to form each dataset. The detailed statistics are shown in Table 1. The class distributions of the AG's News and Yelp are balanced, but are imbalanced on IMDB and Amazon , as shown in Fig. 5. In addition, AG's News is a sentence-level classification dataset, while the others are document-level. Since the AG's News dataset only contains the training and test sets, we randomly sampled 10% of news in the training set for validation. For the other three datasets, we used 80% of samples for training, 10% for validation and the rest 10% for test. In our experiments, the word embeddings were 300-dimensional and initialized by Glove (Pen-nington et al., 2014) 7 . In our comparative experiments, the CNN networks had 400 filters, and their window size was 3. The dimension of LSTM hidden states was 200. The attention query vectors were 200-dimensional. The initial pooling norm p was set to 1, which is consistent with the vanilla attentive pooling. Adam (Kingma and Ba, 2014) was used as the optimizer, and the 7 We do not use language models such as ELMo and BERT since our work focuses on facilitating the pooling technique rather than boosting the performance of our approach against the state-of-the-art methods. batch size was 64. We applied dropout (Srivas-tava et al., 2014) techniques to the word embeddings, CNN networks or LSTMs to mitigate over-fitting, and the dropout ratio was 0.2. These hy-perparameters were tuned on the validation set. In classification tasks the metrics were accuracy and macro-F scores, and in regression tasks the performance was evaluated by rooted mean squared error (RMSE). We reported the average results of 10 independently repeated experiments. We compare the performance of different neural text classification models with different pooling methods to evaluate the performance of our approach. The methods to be compared include: (1) CNN-Avg (Tang et al., 2015b), applying average pooling to the representations learned by CNN to build contextual text representations; (2) CNN-Max (Kim, 2014), using a combination of CNN and max pooling; (3) CNN-Att (Gong and Zhang, 2016), using a combination of CNN and vanilla attentive pooling; (4) CNN-APLN , combining CNN with our APLN approach; (5) LSTM-Last (Hochreiter and Schmidhuber, 1997), using the last hidden state in an LSTM network; (6) LSTM-Avg (Zhao et al., 2016), using average pooling after LSTM; (7) LSTM-Max (Johnson and Zhang, 2016), using max pooling after LSTM; (8) LSTM-Att (Zhou et al., 2016), using attentive pooling after LSTM; (9) LSTM-APLN , combining LSTM with APLN ; (10) HAN (Yang et al., 2016), a hierarchical LSTM network with both word-level and sentence-level attentive pooling; (11) HAN-APLN , using APLN at both word and sentence levels. In methods based on LSTM, we used two parallel LSTMs to scan the input in both directions. The results of these methods are summarized in Table 2, which reveal several findings. First, the methods based on average pooling are usually inferior to those using other pooling methods in our experiments. This is probably because average pooling equally regards different features and cannot distinguish their informativeness. Thus, modeling the importance of different features has the potential to improve text representation learning. Second, the methods based on attentive pooling outperform their variants based on max pooling. This may be because attentive pooling can model the informativeness of contexts for text representation, while max pooling only selects the most salient features, which may be sub-optimal. Third, our APLN approach can consistently outperform other pooling methods, and further hypothesis test results show that the improvement brought by our approach is significant ( p < 0 . 01 ). This may be because vanilla max pooling and attentive pooling methods use a fixed pooling norm for universal text representation learning, and the differences in the characteristics of different tasks and datasets are not considered, which may also be sub-optimal. Our approach can dynamically adapt the pooling norm in different scenarios, which may facilitate text representation learning. In addition, we find the advantage in Macro-F score of our approach over other methods is more significant on the datasets with imbalanced class distributions. This may be because our approach can build text representation in a softer manner, which may help neural models avoid focusing on the clues of major classes only and alleviate their dominance. Fourth, we find hierarchical models ( HAN and HAN-APLN ) outperform flatten models (e.g., LSTM-APLN ) for doc-Methods IMDB Amazon Yelp CNN-Avg 1.388 0.920 0.847 CNN-Max 1.322 0.908 0.834 CNN-Att 1.292 0.899 0.824 CNN-APLN 1.271 0.886 0.801 LSTM-Last 1.316 0.896 0.822 LSTM-Avg 1.343 0.911 0.830 LSTM-Max 1.269 0.890 0.815 LSTM-Att 1.257 0.878 0.799 LSTM-APLN 1.233 0.865 0.784 HAN 1.230 0.866 0.789 HAN-APLN 1.214 0.858 0.776 Table 3: The performance of different methods on rating regression. ument representation learning. This may be because modeling documents in a hierarchical manner can better utilize the structure of documents. In addition, since our approach can be applied at both word and sentence levels in HAN , text representation may be learned more accurately. These results validate the effectiveness of our approach. To further validate the generality of our approach in regression tasks 8 , we also conduct experiments on the IMDB , Amazon and Yelp datasets by formulating the task as a rating regression problem, and the results in terms of RMSE are shown in Table 3. From the results, we find our APLN approach can also bring consistent improvements to many existing methods in the regression task. In this section, we will explore the influence of the scale limiting and re-formulation techniques on the stability and relative pooling speed of our approach. The results are summarized in Table 4. From these results, if the limitation of nonnegativity is removed, the model training is usually unstable, which is intuitive. In addition, if the scale limitation (  1 ) is removed, our model occasionally does not converge. This may be because when p > 1 , our model has the risk of gradient explosion. Thus, the scale of input features should be limited. Besides, the re-formulation method also has critical impacts on our approach. This is probably because directly computing the real-valued 8 We find that the regression labels need to be normalized, or the performance may be sub-optimal. exponents of input features may be numerically unstable. In our approach we decompose the exponents into three stable operations, which is robust to numerical errors. In addition, the pooling speed can be effectively improved, since the computational costs of these atomic operations are usually small. These results validate the effectiveness of our approach. In this section, we study the influence of a small but very important step, i.e., the initialization of the trainable pooling norm p , on the performance of our approach. We compare the performance of LSTM-APLN by varying the initialized values of p . The results are shown in Fig. 6. From Fig. 6, we find the performance of our approach increases when the initialized value of p increases. This is intuitive because when p is too small, the attention network may not be capable of recognizing important contexts effectively, which is not optimal for learning accurate text representations. In addition, when p is initialized with a too large value, the performance will start to decline. This is probably because a large value of p will lead to sharp attentions on critical contexts, and other useful information is not fully exploited. Thus, the performance is also not optimal. These results show that a moderate value (e.g., 1.0) is the most appropriate for initializing the pooling norm p , which is also consistent with standard attentive pooling. In this section, we analyze a critical parameter learned by our model, i.e., the pooling norm p in the APLN module. The evolution of the values of p learned by LSTM-APLN on the four benchmark datasets during model training is portrayed in Fig. 7. From the results, we have several interesting observations. First, the pooling norms learned by our model are consistently less than (cid:3)(cid:57)(cid:68)(cid:79)(cid:88)(cid:72)(cid:3)(cid:82)(cid:73) Figure 6: The influence of the initialization of the pooling norm p on our approach. 1, which indicates that our norm-wise attention is softer than vanilla attention. This may be because L1 norm is not optimal for attentive pooling, and a softer attention manner may be more suitable for learning accurate text representations. Second, we find it is interesting that the norm p consistently decreases when the training epoch increases. This may be because the model may tend to take the global contexts into consideration rather than focus on important ones. Third, a moderate norm p is more appropriate for our approach. This may be because when p is too large, the attentions may be too sparse and useful contextual information is not fully exploited. When p is too small, the attention networks cannot effectively distinguish informative contexts from uninformative ones, which may also be sub-optimal for learning text representations. Fourth, we observe that the norm p learned on datasets with imbalanced class distributions is lower than those with balanced distributions. This may be because on imbalanced dataset, if p is too large, the clues I really liked the case , at first . (b) Attention weights in HAN-APLN . Predicted rating is 3. Figure 8: Visualization of the word-level and sentence-level attention weights in HAN and HAN-APLN on a randomly selected review in the Amazon dataset, whose gold rating score is 3. Darker colors indicate higher attention weights. of the majority classes may be over-emphasized, and other useful information is not fully respected. Thus, the performance of our APLN approach is better when it learns a moderate pooling norm. In this section, we conducted several case studies to further explore the effectiveness of our APLN approach. We visualize the word-level and sentence-level attention weights in HAN and HAN-APLN of a randomly selected review to compare their differences, and the results are portrayed in Fig. 8. According to the results, we have several observations. First, both HAN and HAN-APLN can recognize important words and sentences. For example, the word liked and the sentence I really liked the case, at first. are highlighted since they are important for modeling the opinions condensed by this review. Second, the attentions of HAN are sparse, which indicates that HAN tends to focus more on some contexts in a review such as the first and the third sentence, and pays little attentions to the useful information in other contexts such as the fourth and fifth sentences. In addition, HAN wrongly classifies the rating of this review. This is probably because the rating of a review is usually a synthesis of all opinions conveyed by it. Thus, it may not be optimal for learning accurate text representations if only salient contexts are considered. Third, different from HAN , the attentions of HAN-APLN are smoother. This is probably because the pooling norm learned by our approach is less than 1, which encourages our model to attend to important contexts in a softer manner. In addition, HAN-APLN can classify this review correctly. This is probably because our approach can effectively take global contextual information into consideration, and does not over-emphasize critical contexts. Thus, our APLN approach can learn more accurate text representations than the methods based on vanilla attentive pooling. These results show the effectiveness of our approach. In this paper, we propose an Attentive Pooling with Learnable Norms (APLN) approach for text representation. Instead of using a fixed pooling norm for universal text representation learning, we propose to learn the norm in an end-to-end framework to automatically find the optimal ones for learning text representations in different tasks. In addition, we propose two methods to ensure the numerical stability of the model training. The first one is scale limiting, which limits the scale of input representations to ensure their non-negativity and avoid potential exponential explosion. The second one is re-formulation, which decomposes the exponent operation into several safe atomic operations to avoid computing the real-valued powers of input features with less computational cost. Extensive experiments on four benchmark datasets validate the effectiveness of our approach. In our future work, we will explore several potential directions. First, we plan to explore why the model prefers soft attentions rather than hard ones, which is different from the findings in several prior works based on hard attention. Second, we plan to study how to model the differences on the characteristics of different samples and use different pooling norms, which may have the potential to further improve our approach. Third, we will explore how to generalize our approach to other modalities, such as images, audios and videos, to see whether it can facilitate more attention-based methods. This work was supported by the National Key Research and Development Program of China under Grant number 2018YFC1604002, the National Natural Science Foundation of China under Grant numbers U1936208, U1936216, U1836204, and U1705261.
abstain abstain abstain abstain objective objective objective abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective abstain abstain result other other other other other other other other other other other other other other other other other other other other other method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain result result abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective objective objective abstain abstain abstain objective objective result objective other