This paper explores the problem of ranking short social media posts with respect to user queries using neural networks. Instead of starting with a complex architecture, we proceed from the bottom up and examine the effectiveness of a simple, word-level Siamese architecture augmented with attention-based mechanisms for capturing semantic soft matches between query and post tokens. Extensive experiments on datasets from the TREC Microblog Tracks show that our simple models not only achieve better effectiveness than existing approaches that are far more complex or exploit a more diverse set of relevance signals, but are also much faster. Implementations of our samCNN ( S imple A ttention-based M atching CNN) models are shared with the community to support future work. 1 1 Introduction Despite a large body of work on neural ranking models for traditional ad hoc retrieval over web pages and newswire documents (Huang et al., 2013; Shen et al., 2014; Guo et al., 2016; Xiong et al., 2017; Mitra et al., 2017; Pang et al., 2017; Dai et al., 2018; McDonald et al., 2018), there has been surprisingly little work (Rao et al., 2017) on applying neural networks to searching short social media posts such as tweets on Twitter. Rao et al. (2019) identified short document length, informality of language, and heterogeneous relevance signals as main challenges in relevance modeling, and proposed the first neural model specifically designed to handle these characteristics. Evaluation on a number of datasets from the TREC Microblog Tracks demonstrates state-of-the-art effectiveness as well as the necessity of different model components to capture a multitude of relevance signals. In this paper, we also examine the problem of modeling relevance for ranking short social media posts, but from a complementary perspective. As Weissenborn et al. (2017) notes, most sys-tems are built in a top-down process: authors propose a complex architecture and then validate design decisions with ablation experiments. However, such experiments often lack comparisons to strong baselines, which raises the question as to whether model complexity is empirically justified. As an alternative, they advocate a bottom-up approach where architectural complexity is gradually increased. We adopt exactly such an approach, focused exclusively on word-level modeling. As shown in Figure 1, we examine variants of a simple, generic architecture that has emerged as best practices in the NLP community for tackling modeling problems on two input sequences: a Siamese CNN architecture for learning representations over both inputs (a query and a social media post in our case), followed by fully-connected layers that produce a final relevance prediction (Sev-eryn and Moschitti, 2015; He et al., 2016; Rao et al., 2016), which we refer to as a General Sentence Encoder in Section 2.1. Further adopting best practices, we incorporate query-aware convolutions with an average aggregation layer in the representation learning process. Recently, a number of researchers (Conneau et al., 2017; Mohammed et al., 2018) have started to reexamine simple baselines and found them to be highly competitive with the state of the art, especially with proper tuning. For example, the In-ferSent approach (Conneau et al., 2017) uses a simple BiLSTM with max pooling that achieves quite impressive accuracy on several classifica-tion benchmarks. Our contribution is along similar lines, where we explore simple yet highly effective models for ranking social media posts, to gain insights into querypost relevance matching using standard neural architectures. Experiments with TREC Microblog datasets show that our best model not only achieves better effectiveness than existing approaches that leverage more signals, but also demonstrates 4  speedup in model training and inference compared to a recently-proposed neural model. Our model comprises a representation learning layer with convolutional encoders and another simple aggregation layer. These architectural components are described in detail below. with randomly initialized kernels to learn semantic representations for text. More formally, given query q and post p as sentence inputs, we first convert them to embedding matrices Q and P through an embedding lookup layer, where Q  R n  d and P  R m  d , d is the dimension of embeddings, and n and m are the number of tokens in q and p , respectively. Then we apply a standard convolution operation with kernel window size k over the embedding matrix Q and P . The convolution operation is parameterized by a weight term W  RF  k  d and a bias term b w  RF , where F is the number of convolutional kernels. This generates semantic representation O q  R n  F and O p  R m  F , on which max pooling and an MLP are applied to obtain query representation g q  R d and post representation g p  R d . The weakness of the kernels in the general sentence encoder is that they do not incorporate knowledge from the query when attempting to capture feature patterns from the post. Inspired by attention mechanisms (Bahdanau et al., 2014), we propose two novel approaches to incorporate query information when encoding the post representation, which we introduce below. Query-Aware Attention Encoder (QAtt): In QAtt (Figure 2, left), for each query token, we construct a token-specific convolutional kernel to inject the query information. Unlike methods that apply attention mechanisms after the sentence representations are generated (Bahdanau et al., 2014; Seo et al., 2016), our approach aims to model the representation learning process jointly with an attention mechanism. where U  RF  k  d represents trainable parameters, Q t q is the embedding of token t q with size R d and W t q QAtt  RF  k  d . The element-wise product  is applied between the token embedding Q t q and the last dimension of kernel weights U . In other words, we create F convolutional kernels for each query token, where each kernel is injected with the embedding of that query token via element-wise product. Figure 2 (left) illustrates one kernel for the query token Evernote', where element-wise product is represented by blue dotted arrows. When a QAtt token-specific kernel is applied, a window slides across the post embeddings P and learns soft matches to each query token to generate query-aware representations. On top of the QAtt kernels, we apply max-pooling and an MLP to produce a set of post representations { h i } , with each h i  R d standing for the representation learned from query token t q i . Position-Aware Attention Encoder (PAtt): In the QAtt encoder, token-specific kernels learn soft matches to the query. However, they still ignore positional information when encoding the post semantics, which has been shown to be effective for sequence modeling (Gehring et al., 2017). To overcome this limitation, we propose an alternative attention encoder that captures positional information through interactions between query embeddings and post embeddings. Given a query token t q and the j -th position in post p , we compute the interaction scores by taking the cosine similarity between the word embeddings of token t q and post tokens t p j : j + k  1 from position j to j + k  1 : S j = [ cos ( t q , t p j ); ... ; cos ( t q , t p j + k  1 )] (2) where S j  R k  1 and k is the width of the convolutional kernel we are learning. That is, for each token in the post within the window, we compute its cosine similarity with query token t q . We then convert the similarity vector S j into a matrix:  S j = S j  1 ,  S j  R k  d (3) where 1  R 1  d with each element set to 1. Finally, the PAtt convolutional kernel for query token t q at the j -th position is constructed as: W t q ,j PAtt = V   S j (4) where V  RF  k  d represents the trainable parameters. The element-wise product  is applied between the attention weights  S j and the last two dimensions of kernel weights V . Conceptually, this operation can be thought as adding a soft attention weight (with values in the range of [0 , 1] ) to each convolutional kernel, where the weight is determined by the cosine similarity between the token from the post and a particular query token; since cosine similarity is a scalar, we fill in the value in all d dimensions of the kernel, where d is the size of the word embedding. This is illustrated in Figure 2 (right), where we show one kernel of width two for the query token Evernote'. The brown (green) arrows capture cosine similarity between the query token Evernote' and the first (second) token from the post in the window. These values then serve as weights in the kernels, shown as the hatched areas. Similar to QAtt, the PAtt encoder with max-pooling and an MLP generates a set of post representations { h i } , with each h i standing for the representation learned from query token t q i . It is worth noting that both the QAtt and PAtt encoders have no extra parameters over a general sentence encoder. However, incorporating the query-aware and position-aware information enables more effective representation learning, as our experiments show later. The QAtt and PAtt encoders can also be used as plug-in modules in any standard convolutional architecture to learn query-biased representations. After the representation layer, a set of vectors { g q , g p , { h i }} is obtained. Because our model yields different numbers of h i with queries of different lengths, further aggregation is needed to output a global feature v . We directly average all vectors v = 1 N q (cid:80) h i as the aggregated feature, where N q is the length of the query. To obtain a final relevance score, the feature vectors g q , g p , and v are concatenated and fed into an MLP with ReLU activation for dimensionality reduction to obtain o , followed by batch normalization and fully-connected layer and softmax to output the final prediction. The model is trained end-to-end with a Stochastic Gradient Decent optimizer using negative log-likelihood loss. Datasets and Hyperparameters. Our models are evaluated on four tweet test collections from the TREC 20112014 Microblog (MB) Tracks (Ou-nis et al., 2011; Soboroff et al., 2012; Lin and Efron, 2013; Lin et al., 2014). Each dataset contains around 5060 queries; detailed statistics are shown in Table 1. As with Rao et al. (2019), we evaluated our models in a reranking task, where the inputs are up to the top 1000 tweets retrieved from bag of words ranking using query likelihood (QL). We ran four-fold cross-validation split by year (i.e., train on three years' data, test on one year's data) and followed Rao et al. (2019) for sampling validation sets. For metrics, we used average precision (AP) and precision at rank 30 (P30). We conducted Fisher's two-sided, paired randomization tests (Smucker et al., 2007) to assess statistical significance at p < 0 . 05 . The best model hyperparameters are shown in Table 2. Baselines. On top of QL, RM3 (Abdul-Jaleel et al., 2004) provides strong non-neural results using pseudo-relevance feedback. We also compared against MP-HCNN (Rao et al., 2019), the first neural model that captures specific characteristics of social media posts, which improves over many previous neural models, e.g., K-NRM (Xiong et al., 2017) and DUET (Mitra et al., 2017), by a significant margin. To the best of our knowledge, Rao et al. (2019) is the most effective neural model to date. We compared against two variants of MP-HCNN; MP-HCNN+QL includes a linear interpolation with QL scores. QL on TREC 2013 (queries 111170). Table 3 shows the effectiveness of all variants of our model, compared against previous results copied from Rao et al. (2019). Model 1 illustrates the effectiveness of the basic BiCNN model with a kernel window size of two; combining different window sizes (Kim, 2014) doesn't yield any improvements. It appears that this model performs worse than the QL baseline. Comparing Model 2 to Model 1, we find that query-aware kernels contribute significant improvements, achieving effectiveness comparable to the QL baseline. With Model 3, which captures positional information with the position-aware encoder, we obtain competitive effectiveness compared to Model 8, the full MP-HCNN model that includes interpolation with QL. Note that Model 8 leverages additional signals, including URL information, character-level encodings, and external term features such as tfidf. With Model 4, which interpolates the position-aware encoder with QL, we obtain state-of-the-art effectiveness. Per-Query Analysis. In Figure 3, we show per-query AP differences between the PAtt model and the QL baseline on the TREC 2013 dataset. As we can see, PAtt improves on most of the queries. For the best-performing query 164  lindsey vonn sidelined , we project the hidden states o into a low-dimensional space using t-SNE (Maaten and Hinton, 2008), shown in Figure 4. We observe that with the basic BiCNN model (left), relevant posts are scattered. With the addition of an attention mechanism (either QAtt in the middle or PAtt on the right), most of the relevant posts are clustered together and separated from the non-relevant posts. With PAtt, there appears to be tighter clustering and better separation of the relevant posts from the non-relevant posts, giving rise to a better ranking. We confirmed similar behavior in many queries, which illustrates the ability of our position-aware attention encoder to learn better query-biased representations compared to the other two models. For the worst-performing query 125  Oscars snub Affleck , the PAtt model lost 0.47 in AP and 0.11 in P30. To diagnose what went wrong, we sampled the top 30 posts ranked by the PAtt model and counted the number of posts that contain different combinations of the query terms in Table 4. The PAtt model indeed captures matching patterns, mostly on Oscars and Affleck . However, from the relevance judgments we see that snub is the dominant term in most relevant posts, while Oscars is often expressed implicitly. For example, QL assigns more weight to the term snub in the relevant post  argo wins retributions for the snub of ben affleck  because of the term's rarity; in contrast, the position-aware encoder places emphasis on the wrong query terms. Model Performance. Finally, in terms of training and inference speed, we compared the PAtt model with MP-HCNN on a machine with a GeForce GTX 1080 GPU (batch size: 300). In addition to being more effective (as the above results show), PAtt is also approximately 4  faster. In this paper, we proposed two novel attention-based convolutional encoders to incorporate query-aware and position-aware information with minimal additional model complexity. Results show that our model is simpler, faster, and more effective than previous neural models for searching social media posts. This research was supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada.
objective objective result abstain other abstain abstain other abstain method abstain abstain objective objective abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method method abstain abstain abstain abstain method method method abstain result abstain abstain abstain result abstain result abstain result result result result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain objective result other