j.e.weeds@sussex.ac.uk Abstract Paraphrase identification involves identifying whether a pair of sentences express the same or similar meanings. While cross-encoders have achieved high performances across several benchmarks, bi-encoders such as SBERT have been widely applied to sentence pair tasks. They exhibit substantially lower computation complexity and are better suited to symmetric tasks. In this work, we adopt a bi-encoder approach to the paraphrase identification task, and investigate the impact of explicitly incorporating predicate-argument information into SBERT through weighted aggregation. Experiments on six paraphrase identification datasets demonstrate that, with a minimal increase in parameters, the proposed model is able to outperform SBERT/SRoBERTa significantly. Further, ablation studies reveal that the predicate-argument based component plays a significant role in the performance gain. Paraphrases are sentences that express the same or similar meanings with different wording (Bhagat and Hovy, 2013). Paraphrase pairs are either fully or largely semantically equivalent. For example: a) Marriage equality law passed in Rhode Island b) Rhode Island becomes the 10th state to enact marriage equality It is generally considered to be a symmetric task where the paraphrase relation holds in both directions (Bhagat and Hovy, 2013; Yang et al., 2019). Since word order and sentence structure are crucial in determining sentence meaning, effective paraphrase models must be structure-aware and word order sensitive. In light of this, paraphrase datasets have been created that are specifically designed to encourage models to consider structural differences (Xu et al., 2015; Zhang et al., 2019b). For example, PIT2015 (Xu et al., 2015) consists of paraphrase pairs that are lexically diverse and non-paraphrase pairs that are lexically similar but semantically dissimilar. There are generally two pre-trained based approaches for sentence pair tasks such as paraphrase identification. The first is the cross-encoder approach, which involves concatenating the two input sentences and performing full-attention over the input. The second is the bi-encoder approach, which adopts a conjoined twin network structure and maps each sentence onto separate representations, which can then be compared using similarity measures such as cosine. Though typical cross-encoders like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b) have set state-of-the-art performance on various sentence pair tasks (Zhang et al., 2021; Xia et al., 2021), they still face challenges from both extreme computational overhead for many use cases (Reimers and Gurevych, 2019; Thakur et al., 2021) and inconsistent predictions (ranging from 2.66% to 8.46% depending on specific datasets) when dealing with symmetric tasks (Chen et al., 2020). In contrast, a bi-encoder approach such as Sentence-BERT (SBERT) (Reimers and Gurevych, 2019) encodes sentences separately and generates high-quality embeddings for each of them. This architecture enables sentence embeddings to be pre-computed, supporting efficient indexing and comparison between different sequences. Due to the nature of bi-encoders, the symmetry property will be preserved as long as no asymmetry is introduced in subsequent layers. These properties make bi-encoders appealing for the paraphrase identification task. Accordingly, here, we focus on bi-encoders rather than cross-encoders. very simple strategy, which is mean-pooling over all tokens, to generate sentence embeddings. As previously discussed, models should ideally be sensitive to any structural differences between two sentences. Relational Graph Convolutional Networks (RGCNs) (Schlichtkrull et al., 2018) have been used to introduce structural information (e.g. dependency/semantic parse trees) into SBERT and improvements have been reported on unsupervised similarity comparison tasks (Peng et al., 2021). One drawback of RGCNs is the size of the parameter space. For example, a single-layer RGCN can involve more than 30 million parameters. Furthermore, as we will demonstrate, the performance gain on different paraphrase identification datasets is not consistent. An important aspect of sentence meaning concerns its predicate-argument structure. This has been utilised to generate paraphrases (Kozlowski et al., 2003) and to compare sentence meanings (Shan et al., 2009). Inspired by the Self-Explain model (Sun et al., 2020) which uses a span-based framework to generate sentence embeddings, we propose a method that effectively introduces sentence structure into SBERT via the aggregation of predicate-argument spans. This self-attention based aggregation allows us to gain benefits with minimal increased cost in terms of additional parameters. Empirical results indicate that the proposed model yields improvements on six benchmarks for paraphrase identification. Upon closer investigation, we find the predicate-argument span (PAS) component plays a crucial role in the performance gains and can be easily generalised to other models. The problem of paraphrase identification has been explored now for several decades (Mihalcea et al., 2006; Kozareva and Montoyo, 2006). Prior to the emergence of pre-trained models, bi-encoder structures were widely used. For example, Mueller and Thyagarajan (2016) applied LSTM in a twin architecture with tied weights and used Manhattan distance to compute similarity. InferSent (Con-neau et al., 2017) exploited BiLSTM in a similar twin structure with a fully-connected layer for classification over interacted sentence embeddings. Although their model was mainly proposed for transfer learning, experiments showed that it achieves good performance when directly trained on in-domain data. Some bi-encoders do not generate single-vector sentence embeddings and allow direct comparisons between the words in the two sentences. Pang et al. (2016) proposed MatchPyramid where interaction matrix is constructed, and convolutional networks were used to extract features for final classification. PMWI (He and Lin, 2016) introduced more fine-grained comparisons between words to better dissect the meaning difference. ESIM (Chen et al., 2017) further utilised BiLSTM to bring contextualised token representations and allow rich interactions between tokens. Researchers have further improved these models by incorporating context and structure information (Liu et al., 2019a), as well as character-level information (Lan and Xu, 2018). After the emergence of pre-trained models, cross-encoders like BERT and RoBERTa have achieved state-of-the-art performance on various sentence pair tasks including paraphrase identification. Zhang et al. (2019a) introduced pairwise word interaction mechanism into BERT. Zhang et al. (2021) improved BERT on paraphrase tasks by using CNNs to gather local information and an auxiliary task to further bring in semantic relation information. Xia et al. (2021) injected similarity matrices into BERT's attention mechanism. Though improved performance can be obtained, cross-encoders have known drawbacks. In particular, Reimers and Gurevych (2019) showed the extreme computation overhead of cross-encoders, and Chen et al. (2020) demonstrated that cross-encoders often give inconsistent predictions when reversing the input sentence order. Based on these factors, bi-encoders are often preferred for the paraphrase identification task. Though pre-trained models like BERT seem to encode certain structures in their contextualised representations, open questions remain about how to better utilise such information (Hewitt and Manning, 2019; Clark et al., 2019) and how useful the hidden structure is compared to externally provided sentence structures (Glava and Vuli  c, 2021; Dai et al., 2021). Recent improvements are also observed on various natural language understanding tasks by incorporating structural information into pre-trained models. SentiBERT proposed by Yin et al. (2020) 5580 incorporates constituency parse tree into BERT for sentiment analysis. Xu and Yang (2019) model each sentence as a directed dependency graph by using RGCN, and achieve improvements on pronoun resolution. Zhang et al. (2020) propose a semantics-aware BERT (SemBERT) model by further encoding semantic labels with BERT using a GRU. RGCNs have also been used by Wu et al. (2021) to introduce semantic information into RoBERTa, and achieved consistent improvements when fine-tuned on problem-specific datasets. Peng et al. (2021) propose a SBERT-RGCN model where structural information is explicitly encoded into SBERT in a similar way, achieving improvements on unsupervised sentence similarity comparison tasks. Similar efforts can be seen where researchers try to provide syntax information via self-attention mechanism (Bai et al., 2021; Li et al., 2020). Self-Explain model proposed by Sun et al. (2020) focuses on continuous text spans. It generates sentence embeddings by taking the weighted sum over all possible continuous text spans rather than individual tokens in the sentence. Though, Self-Explain achieves improvements over SentiBERT and SemBERT on sentiment analysis and language inference tasks, the continuous span strategy only captures linear structure and not differences in linguistic structure. In this paper, we draw inspiration from it, designing a similar span-based component to incorporate predicate-argument spans. Our proposed model adopts the same conjoined twin architecture as SBERT and turns focus to the predicate-argument structure of the given sentence. As depicted in Figure 1, the model consists of different components: BERT: Each sentence is first fed into the pretrained BERT-base model to produce both a sentence representation, by applying mean-pooling over all token representations from the last hidden layer, and an original contextualised sequence-length token representation, which is used to derive predicate-argument span representations. Predicate Argument Spans (PAS): We use Al-lenNLP (Gardner et al., 2018) with its BERT-based semantic role labelling (SRL) tagger to obtain predicates and relevant arguments for all input sentences. We group the predicate and its arguments together to generate predicate-argument spans. The initial position in the sentence determines their position in the span. An example of such spans is shown below: He slices tomatoes in the kitchen From this sentence, the predicate is the verb slices , and the three arguments are ( he , tomatoes and in the kitchen ), involving the relations ( ARG0 , ARG1 and ARGM-LOC ), respectively. In this way, we form three predicate-argument spans and split them into individual words: ( He , slices ), ( slices , tomatoes ), ( slices , in , the , kitchen ). One sentence is likely to have multiple predicates, by adopting this strategy, we are able to obtain all potential predicate-argument spans in the given sentence. We further utilise these extracted spans to form a span-based sentence representation. If no predicate-argument structure can be found in the sentence, we directly use the representation after mean-pooling over all tokens as its sentence representation. Aggregation: After obtaining all predicate-argument spans, we derive corresponding span representations by looking at BERT's token representations. In BERT/RoBERTa, tokenization yields sub-tokens, whereas in the created spans, we have an entire word token. To properly align them, we use the same tokenizer to break the original word into sub-tokens and represent it as a sequence of sub-tokens in the span if a sub-token exists. Given a predicate-argument span sequence s = { s 1 , s 2 , ..., s N } in the sentence, where N denotes the number of spans and every span s i consists of tokens { x 1 , ..., x l } that make up the span. For each span s i , we obtain its dense vector representation h i by taking mean-pooling over all tokens in it: h i = MeanP ooling ( x 1 , .., x l ) (1) Therefore, the whole representation for span sequence s is represented as h = { h 1 , h 2 , ..., h N } , where h i  RD . Then, we aggregate information from all spans using a simple self-attentive mechanism. Following Sun et al. (2020), this is achieved by first assigning weights  i to each span h i and combining these representations using weighted sum: o i = W  h i + b  i = exp ( o i ) N (cid:80) j =1 exp ( o j ) (2) 5581 Figure 1: The proposed model in twin structure. where W  R 1  D and b are learnable parameters. The span-based sentence representation  h from the aggregation component is the weighted average of all predicate-argument span representations:  h = N (cid:88) i =1  i  h i (3) The weights are learned during training. This gives the model flexibility to decide the best combination method on its own. The combination of self-attentive mechanism and predicate-argument spans allow us to construct structure-aware sentence embeddings without introducing a large number of parameters. Connect BERT and Aggregation: The final sentence representation is the concatenation of both BERT mean-pooling based sentence representation and the span-based sentence representation. Sentence embeddings of the given sentence-pair are then combined using vector operations before passing to the final classifier for training as shown in Figure 1. To combine the embeddings, we use the concatenation of the element-wise multiplication u  v and the absolute element-wise difference | u  v | . This is different to the typical concatenation strategy used with SBERT/SRoBERTa (Reimers and Gurevych, 2019) which introduces asymmetry into the task by using (u, v, |u-v|). In initial experiments, we tested the prediction consistency of SBERT on paraphrase tasks and found that, across different datasets, between 2.78% and 9.16% of test predictions change when the sentence order is reversed. Furthermore, here, we find that SBERT performs worse on paraphrase tasks with (u, v, |u-v|) compared to (|u-v|, u  v ). Results are given in Table 5 and discussed in Section 5.1. Finally, we note that in this twin structure, all parameters are shared and are updated accordingly. Cross-entropy loss is used for optimisation. We compare our model with SBERT, SRoBERTa 1 and the SBERT-RGCN (Peng et al., 2021) which utilises RGCN to incorporate structures into SBERT with an introduction of 32 million extra parameters 2 . The original sentence-pair aggregation strategy of these models is (u, v, |u-v|). We modify this to (|u-v|, u  v ) as discussed in Section 3, but we retain the original notation. We adopt their structures and directly fine-tune the whole model on downstream tasks from the original BERT/RoBERTa checkpoints. We considered two strategies to apply SBERT on classification inference. One involved finding the optimal similarity threshold on the development set and then applying it on the test set, while the other involved directly using the trained classifier. In this paper, 1 https://github.com/UKPLab/sentence-transformers. Due to limited computational resources, all pre-trained models are of base size. 2 SBERT-RGCN tried both dependency and semantic parse trees. We evaluate our model on six binary paraphrase identification benchmarks. The statistics of these datasets are listed in Table 1. Below we give some basic descriptions:  Microsoft Research Paraphrase Corpus (MSRP) : A corpus of sentence pairs obtained by clustering news articles with an SVM classifier and human annotations (Dolan and Brockett, 2005). It has 4,076 train data and 1,725 test data. In this paper, we split 10% of training data as the validation set according to GLUE (Wang et al., 2019) standardised splits. TwitterURL : To better study the realistic language usage, Lan et al. (2017) proposed the TwitterURL corpus where sentence pairs in the dataset are collected from tweets that share the same URL of news articles. PIT2015 : The corpus is derived from Twit-ter's trending topic data, containing 18,763 sentence pairs on more than 400 distinct top-ics (Xu et al., 2015). Given we are dealing with binary classification, we discard debatable sentence pairs according to its guideline and obtain 16,510 sentence pairs in total. This dataset contains paraphrase pairs that are lexically diverse and non-paraphrase pairs that are lexically similar, but semantically dissimilar. To capture these properties, models are assumed to be structure-aware. Quora Question Pairs (QQP) : The Quora Question Pairs dataset is a collection of potential duplicate question pairs from the QA website Quora.com (Iyer et al., 2017). In this paper, we adopt the same split strategy as in Wang et al. (2017). PAWS_QQP : QQP is criticised for lacking negative examples with high lexical overlapping. Models trained on QQP tend to mark any sentence pairs with a high word overlap as paraphrases despite clear clashes in meaning. In light of these factors, Zhang et al. (2019b) proposed a new paraphrase identification dataset which has extremely high lexical overlap by applying word scrambling and back translation to sentences in QQP. PAWS_Wiki : Similar to PAWS_QQP, Zhang et al. (2019b) applied the same technique on sentences obtained from Wikipedia articles to construct sentence pairs. Both PAWS datasets aim to measure sensitivity of models on word order and sentence structure. Due to the lack of development set for PAWS_QQP, we use PAWS_Wiki's development set for early stopping since they are constructed in the same way. It is worth noting that both PIT2015 and PAWS_QQP datasets have relatively small test sets compared to others. Following the SBERT training protocol, we train all models with a batch-size of 16. We tune the learning rate in the range of (1e-5, 2e-5, 5e-5) with Adam optimizer and a linear learning rate warmup over 10% of the training data. All models are trained for four epochs and use the development set for early stopping with a patience of 5. The evaluation step depends on actual tasks but roughly we evaluate them on the development set twice each epoch. The maximum sequence length is set to be 128. All experiments are conducted on NVIDIA Titan V GPUs. The main experiment results are summarised in Table 2. We report the averaged F1 score of positive class with standard error. In the table, we see that the proposed model consistently outperforms its SBERT and SRoBERTa versions on 5 paraphrase identification tasks and show competitive, but not statistically significantly different results on QQP. As also revealed by Zhang et al. (2019b), negative examples in QQP often have low lexical overlap, and models trained on it tend to mark any sentence pairs with high word overlap as paraphrases. We 5583 QQP TwitterURL MSRP PAWS_Wiki PAWS_QQP PIT2015 SBERT 90.780.09 70.850.28 81.670.46 81.570.53 66.010.45 52.031.44 SBERT-RGCN 90.410.09 70.400.22 81.700.17 81.140.81 66.220.75 59.110.93 PAS+SBERT 90.740.06 72.120.26 83.420.23 82.600.18 68.850.73 59.191.85 SRoBERTa 90.790.09 70.690.23 81.690.53 81.420.93 67.350.97 52.672.75 PAS+SRoBERTa 90.760.03 72.040.23 83.220.46 82.870.35 69.680.72 59.502.74 Table 2: Results on six paraphrase identification tasks, we calculate the F1 score of the positive class given most of them are imbalanced datasets. reason that the QQP task is relatively easy and does not require much structural information to achieve high scores. For tasks like PAWS_QQP and PIT2015 where structures are more important, the performance gap is more apparent. Furthermore, despite bringing in more than 30 million parameters and explicitly encoding sentence structures with a complex model, SBERT-RGCN does not significantly outperform SBERT on most of these tasks (excluding PIT2015) and underperforms our proposed model. In summary, the proposed model shows improved performances on five out of six paraphrase tasks, demonstrating the advantages of bringing in the predicate-argument structure. Moreover, when we combine PAS with SRoBERTa, we get similar performance gains, proving the generalisation ability of our component. Similarly in Reimers and Gurevych (2019), we only observe minor differences by replacing SBERT with SRoBERTa. The number of parameters for different approaches are shown in Table 3. We note that compared to SBERT, our proposed model introduces 3,840 additional parameters, and if we only consider the span-based component, only 768 additional parameters are introduced. In comparison, SBERT-RGCN brings in more than 32 million parameters. In order to better understand how the performance gain is achieved, we have carried out several experiments to investigate different aspects of the proposed model. The following experiments are conducted only with SBERT, since we would expect similar results with SRoBERTa. Our proposed model is made of different components and so it is important to dissect the impact of each component so as to explain the improved performance. Given that the final sentence representation is the concatenation of both mean-pooling based BERT representation and the weighted sum of span representations, we first assess their performances individually on six datasets. Furthermore, it is necessary to assess the impact of adopting the weighted sum strategy when we derive span-based sentence representations. We experimented with simple averaging over all spans and compared it with the weighted sum where the model learns to combine different spans. The ablation experiment results are shown in Table 4. The SBERT-only component appears to perform the poorest, and the complete model achieves the highest performance on five out of six tasks. By only using the span-based sentence representation, we are able to achieve significant improvements over SBERT on most of these tasks. The improvements are more substantial when concatenating with SBERT sentence representations. We observe considerable performance decreases on most tasks when switching from weighted sum to simple averaging, which further verifies the benefits of adopting learnable weights. The original asymmetric sentence aggregation strategy (u, v, |u-v|) of SBERT assumes an ordering of the sentences by concatenating two individual 5584 QQP TwitterURL MSRP PAWS_Wiki PAWS_QQP PIT2015 PAS+SBERT 90.740.06 72.120.26 83.420.23 82.600.18 68.850.73 59.191.85 SBERT-only 90.780.09 70.850.28 81.670.46 81.570.53 66.010.45 52.031.44 PAS only 90.700.08 71.640.14 82.910.12 82.260.34 67.380.22 54.951.45 PAS only (simple average) 90.110.13 71.090.30 82.130.14 81.850.26 66.550.41 51.821.31 Table 4: Experimental results for ablation study. sentence embeddings. u  v has been widely used elsewhere (Conneau et al., 2017; Cer et al., 2018) and we found that concatenating this with |u-v| gave the best performance on all tasks. The results are summarised in Table 5. Therefore, we use (|u-v|, u  v ) as our concatenation method for all of our other experiments. The impact of incorporating predicate-argument spans into SBERT in terms of the performance on various paraphrase identification tasks has been investigated in the above experiments. We now address the question of whether it is the use of specifically predicate-argument based spans that is critical, or whether this is simply a result of the fact that we are benefiting from the use of representations based on spans rather than all tokens. To verify this, we further conduct experiments with different span strategies. We pick three paraphrase identification datasets for this purpose (MSRP, PAWS_QQP and PIT2015) since performance gaps between PAS+SBERT and SBERT are more apparent in previous experiments. Here we experiment with two other span strategies. The first, inspired by the Self-Explain model (Sun et al., 2020), is the continuous random span, where instead of following the predicate-argument structure, we randomly sample continuous word sequences from the sentence to build a span. The length of the sampled spans is arbitrary. To make a fair comparison, the number of sampled spans is Task Span Type Span only Self-Explain* SBERT MSRP PAS 82.910.12 81.230.27 81.670.46 ContinuousRandomSpan 81.400.43 Random Span 81.860.47 PAWS_QQP PAS 67.380.22 66.880.46 66.010.45 ContinuousRandomSpan 65.450.44 Random Span 65.750.74 PIT2015 PAS 54.951.45 47.601.01 52.031.44 ContinuousRandomSpan 51.621.92 Random Span 50.852.11 Table 6: Evaluation for different span strategies using our span-only component on three datasets. the same as that of the predicate-argument spans in the sentence. The other one is random span, where we do not necessarily sample continuous words, but allow word leaps from one to another. In this strategy, we have the opportunity to get both continuous and discontinuous word sequences to form spans, which better matches the scenario of PAS. The only difference between these two strategies and PAS is the words in the span. We also experiment with a bi-encoder approach more directly based on the Self-Explain cross-encoder model (Sun et al., 2020). This model extracts all possible continuous text spans and obtains span representations by taking the first and last token in the span, passing them through a complex mapping function. Unlike our PAS model, this 5585 10 30 60 100 percentage of training data (%) 62 64 66 68 70 72 SBERTPAS+SBERT compared to SBERT. Table 6 shows the results. In order to focus on the impact of different span strategies, we only use the PAS component and do not concatenate it with SBERT sentence representations in this experiment. As shown in the table, the PAS-based model outperforms the Self-Explain inspired bi-encoder model and achieves the best performance among all other span-based models. The continuous random span and the random span model have comparable performances with SBERT. This is expected because they do not introduce linguistically-meaningful structures and the impact of contextu-alisation makes them similar to SBERT despite the absence of some tokens. Despite introducing 2.36 million more parameters, the Self-Explain inspired bi-encoder model does not show consistent improvements over SBERT on these datasets, which further suggests the importance of the predicate-argument structure in this paraphrase identification task. In order to examine the stability of our model and the impact of the predicate-argument structure when different sizes of training data are available, we conduct experiments with different training data scales. We randomly sample from 10% to 100% data (10%, 30%, 60%, 100%) from the training set as training data. We show the results in Figure 2. In spite of limited increased parameters, the proposed model appears to yield consistent improvements across different training scales. We also note that, whilst our proposed model performs comparably to SBERT on QQP when trained with the complete data-set, we can see that when only a small proportion of training data (e.g. 10%, 30%) is available, our model demonstrates improvements over SBERT. Thus the introduction of predicate-argument structures may be more beneficial with limited annotated training data. In this work, we propose a method which effectively introduces sentence structure to a sentence embedding via the aggregation of predicate-argument spans (PAS). Experiments with SBERT and SRoBERTa show that such method brings improvements on six paraphrase identification tasks. Compared to models based on RGCNs, our method obtains more consistent benefits with minimal increased cost in terms of numbers of parameters. Upon closer investigation, we show that the PAS component and its learnable weights play a substantial impact in the performance gain. This PAS component, as demonstrated with SRoBERTa, can be easily extended to other models that require the generation of sentence embeddings. Our future work will include enhancing the structural difference between sentences by taking use of the argument tag information. We thank all anonymous reviewers for their insightful comments, and NVIDIA for the donation of the GPU that supported our work. Also, we would like to thank Bowen Wang for helpful discussions and proofreading.
abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain result other other other other other other other other other other other other other other other other other other other other other other other other other other other other method abstain abstain abstain method abstain abstain method abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain objective abstain result result abstain method other other