Recently, finetuning a pretrained language model to capture the similarity between sentence embeddings has shown the state-of-the-art performance on the semantic textual similarity (STS) task. However, the absence of an interpretation method for the sentence similarity makes it difficult to explain the model output. In this work, we explicitly describe the sentence distance as the weighted sum of contextualized token distances on the basis of a transportation problem, and then present the optimal transport-based distance measure, named RCMD; it identifies and leverages semantically-aligned token pairs. In the end, we propose CLRCMD, a contrastive learning framework that optimizes RCMD of sentence pairs, which enhances the quality of sentence similarity and their interpretation. Extensive experiments demonstrate that our learning framework outperforms other baselines on both STS and interpretable-STS benchmarks, indicating that it computes effective sentence similarity and also provides interpretation consistent with human judgement. The code and checkpoint are publicly available at https: //github.com/sh0416/clrcmd . Predicting the semantic similarity between two sentences has been extensively studied in the literature (Gomaa et al., 2013; Agirre et al., 2015; Ma-jumder et al., 2016; Cer et al., 2017). Several recent studies successfully utilized a pretrained language model such as BERT (Devlin et al., 2019) by finetuning it to capture sentence similarity (Reimers and Gurevych, 2019). To be specific, they define a similarity score between sentence embeddings, which are obtained by aggregating contextualized token embeddings (e.g., avg pooling) or using a special token (e.g., [CLS] ), then optimize the score  This work was done during internship at Scatterlab. for natural language inference (NLI) or semantic textual similarity (STS) tasks (Gao et al., 2021). Along with the quality of sentence similarity, interpreting the predicted sentence similarity is also important for end-users to better understand the results (Agirre et al., 2016; Gilpin et al., 2018; Rogers et al., 2020). In general, finding out the cross-sentence alignment and the importance of each aligned part is useful for analyzing sentence similarity (Sultan et al., 2015). For example, there were several attempts to use explicit features (e.g., TF-IDF) for easily analyzing the interaction among the shared terms (Salton and Buckley, 1988) or to adopt sophisticated metrics (e.g., word mover's distance) for explicitly describing it by the importance and similarity of word pairs across two sentences (Kusner et al., 2015). However, for recent approaches that leverage sentence embeddings from a pretrained model, it has not been studied how the cross-sentence interaction of each part contributes to the final sentence similarity. In this work, we propose an analytical method based on optimal transport to analyze existing ap-5969 proaches that leverage a pretrained model. We consider a sentence similarity measure a solution to a transportation problem, which aims to transport a collection of contextualized tokens in a sentence to the ones in another sentence. As byproducts of the problem, we obtain a cost matrix and a transportation matrix, which encode the similarities of all token pairs across sentences and their contributions to the sentence similarity, respectively. Using this analytical method, we point out that the existing approaches suffer from the rank-1 constraint in the transportation matrix; this eventually keeps the model from effectively capturing the similarities of semantically-aligned token pairs into sentence similarity. For example, considering transportation in a contextualized embedding space (Figure 1), the distance between averaged token embeddings (or-ange arrows) cannot clearly represent the distance of semantically-aligned token pairs (blue arrows). To resolve the above limitation and enhance the interpretability of a model, we present a novel distance measure and a contrastive learning framework that optimizes the distance between sentences. First, we apply optimal transport in a contextualized embedding space and leverage the optimal solution for a relaxed transportation problem as our distance measure. This sentence distance is composed of the distances of semantically-aligned token pairs; this makes the result easily interpretable. Furthermore, we present a contrastive learning framework that adopts the proposed distance to finetune the model with token-level supervision. It optimizes the model to learn the relevance of semantically-aligned token pairs from that of sentence pairs, which further enhances interpretability. We extensively evaluate our approach and validate the effectiveness of its sentence similarity and interpretation. The comparison on 7 STS benchmarks supports the superiority of sentence similarity predicted by the model trained by our framework. In particular, the evaluation on 2 interpretable-STS datasets demonstrates that the proposed distance measure finds out semantically relevant token pairs that are more consistent with human judgement compared to other baseline methods. Our qualitative analysis shows that both the token alignment and their similarity scores from our model serve as useful resources for end-users to better understand the sentence similarity. Most recent studies tried to leverage a pretrained language model with various model architectures and training objectives for STS tasks, achieving the state-of-the-art performance. In terms of model architecture, Devlin et al. (2019) focus on exhaustive cross-correlation between sentences by taking a concatenated text of two sentences as an input, while Reimers and Gurevych (2019) improve scalability based on a Siamese network and Humeau et al. (2020) adopt a hybrid approach. Along with the progress of model architectures, many advanced objectives for STS tasks were proposed as well. Specifically, Reimers and Gurevych (2019) mainly use the classification objective for an NLI dataset, and Wu et al. (2020) adopt contrastive learning to utilize self-supervision from a large corpus. Yan et al. (2021); Gao et al. (2021) incorporate a parallel corpus such as NLI datasets into their contrastive learning framework. Despite their effectiveness, the interpretability of the above models for STS tasks was not fully explored (Belinkov and Glass, 2019). One related task is interpretable STS, which aims to predict chunk alignment between two sentences (Agirre et al., 2016). For this task, a variety of supervised approaches were proposed based on neural networks (Konopk et al., 2016; Lopez-Gazpio et al., 2016), linear programming (Tekumalla and Jat, 2016), and pretrained models (Maji et al., 2020). However, these methods cannot predict the similarity between sentences because they focus on finding chunk alignment only. To the best of our knowledge, no previous approaches based on a pretrained model have taken into account both sentence similarity and interpretation. Optimal transport (Monge, 1781) has been successfully applied to many applications in natural language processing (Li et al., 2020; Xu et al., 2021), by the help of its ability to find a plausible correspondence between two objects (Lee et al., 2021a,b). For example, Kusner et al. (2015) adopt optimal transport to measure the distance between two documents with pretrained word vectors. Zhao et al. (2019) adopt optimal transport for evaluating text generation and Zhang et al. (2020) take a greedy approach leveraging pretrained language model. In addition, Swanson et al. (2020) discover 5970 the rationale in text-matching via optimal transport, thereby improving model interpretability. One well-known limitation of optimal transport is that finding the optimal solution is computationally intensive, and thus approximation schemes for this problem have been extensively researched (Grauman and Darrell, 2004; Shirdhonkar and Jacobs, 2008). To get the solution efficiently, Cuturi (2013) provides a regularizer inspired by a probabilistic theory and then uses Sinkhorn's algorithm. Kusner et al. (2015) relax the problem to get the quadratic-time solution by removing one of the constraints, and Wu et al. (2018) introduce a kernel method to approximate the optimal transport. We first analyze the similarity measure used by existing models from the perspective of a transportation problem. Considering the above analysis, we present a novel distance measure and a contrastive sentence learning framework to enhance the interpretability of a sentence similarity model. We briefly explain the transportation problem and how to interpret the total transportation cost as a distance measure. A transportation problem consists of three components: states before and after transportation, and a cost matrix. In general, the two states are represented in high-dimensional simplex, i.e., d 1   d 1 and d 2   d 2 , where each dimension implies a specific location with a nonnegative quantity. The cost matrix M  R d 1  d 2 encodes the unit transportation cost from location i to j into M i,j . In this situation, we search the transportation plan to transport from d 1 to d 2 with the minimum cost. Using the above notations, the optimization problem is written as follows: minimize T  R d 1  d 2  0 (cid:88) i,j T i,j M i,j (1) subject to T (cid:62) (cid:126) 1 = d 2 , T (cid:126) 1 = d 1 , where each entry of the transportation matrix T i,j indicates how much quantity is transferred from location i to j . The optimal solution to this problem is called optimal transport, which is also known as earth mover's distance (EMD): d EMD M := (cid:88) i,j T  i,j M i,j . In Equation (2), the distance is computed by the sum of element-wise multiplications of the optimal transportation matrix T  and the cost matrix M . In this sense, EMD considers the optimality of distance when combining unit costs in M . That is, the priority of each unit cost when being fused to the distance is encoded in the transportation matrix, which serves as a useful resource for analyzing the distance. We express cosine similarity with average pooling as a transportation problem and analyze its properties in terms of the transportation matrix. Note that this similarity measure is widely adopted in most of the previous studies (Reimers and Gurevych, 2019; Wu et al., 2020; Gao et al., 2021). Formally, for a sentence of length L , the sentence embedding is generated by applying average pooling to L contextualized token embeddings, i.e., s = 1 L (cid:80) L i =1 x i , where x i is the i -th token embedding obtained from a pretrained model. Using the sentence embeddings, the sentence similarity is defined by s AVG = cos( s 1 , s 2 ) = s 1 (cid:62) s 2 (cid:107) s 1 (cid:107)(cid:107) s 2 (cid:107) . This average pooling-based sentence similarity can be converted into the distance, d AVG = 1  s AVG , described by the token embeddings as follows: d AVG = 1  L 1 (cid:88) i =1 L 2 (cid:88) j =1 1 L 1 L 2 (cid:107) x 1 i (cid:107)(cid:107) x 2 j (cid:107) (cid:107) s 1 (cid:107) (cid:107) s 2 (cid:107) x 1 i (cid:62) x 2 j (cid:107) x 1 i (cid:107)(cid:107) x 2 j (cid:107) . From the perspective of Equation (1), this distance is interpreted as a naive solution of a special transportation problem, where the cost matrix and the transportation matrix are MAVG i,j = (cid:107) s 1 (cid:107)(cid:107) s 2 (cid:107) (cid:107) x 1 i (cid:107)(cid:107) x 2 j (cid:107)  cos( x 1 i , x 2 j ) , TAVG i,j = 1 L 1 L 2 (cid:107) x 1 i (cid:107)(cid:107) x 2 j (cid:107) (cid:107) s 1 (cid:107) (cid:107) s 2 (cid:107) . (3) Each entry of the cost matrix includes negative cosine similarities between token embeddings, and the contribution of each token pair to the sentence distance (i.e., the transportation matrix) is determined by the norms of the token embeddings. In theory, the rank of the transportation matrix is constrained to be one, which prevents effective integration of the token distances into the sentence distance. In practice, it is impossible to involve only 5971 semantically-aligned token pairs across sentences because all possible token pairs are considered by the products of their norms. From this analysis, we point out that the average pooling-based similarity is not effective enough to capture the token correspondence between sentences. To resolve the ineffectiveness of the existing measure, we introduce a novel distance measure based on optimal transport. We first define a transportation problem that considers semantic relevance in a contextualized embedding space. Given the token embeddings of two sentences from a pretrained language model, we construct a cost matrix MCMD  RL 1  L 2 that encodes token similarities using cosine distance, and define the state vectors for the two sentences as one vectors normalized by their sentence lengths d 1 := 1 L 1 (cid:126) 1 and d 2 := 1 L 2 (cid:126) 1 . As discussed in Section 3.1, we consider the optimal solution to this problem as a distance measure named contextualized token mover's distance (CMD): MCMD i,j := 1  cos (cid:0) x 1 i , x 2 j (cid:1) , d CMD M := (cid:88) i,j T  i,j MCMD i,j . However, finding T  incurs huge computational complexity of O ( L 3 log L ) where L = max( L 1 , L 2 ) (Villani, 2008). For this reason, we relax the optimization problem by removing the first constraint, T (cid:62) (cid:126) 1 = d (cid:48) , similar to Kusner et al. (2015). The optimal solution for this relaxed transportation problem is found in O ( L 2 ) , keeping the rank of the transportation matrix larger than one. In the end, the optimal transportation matrix and the corresponding distance named relaxed CMD (RCMD) are derived as follows: TRCMD 1 i,j = (cid:40) 1 L 1 if j = argmin j (cid:48) MCMD i,j (cid:48) 0 otherwise, d RCMD 1 M := 1 L 1 (cid:88) i min j MCMD i,j . (4) Similarly, the elimination of the second constraint, T (cid:126) 1 = d , results in TRCMD 2 and d RCMD 2 M , where the solutions for the two relaxed problems use min operation on the cost matrix in a row-wise and a column-wise manner, respectively. Note that TRCMD 1 represents the token-level binary alignment from the first sentence to the second sentence and accordingly the final distance is computed by averaging all the distances of the aligned token pairs. Also, it is obvious that TRCMD 1 has a much higher rank than TAVG , which implies that it can express more complex token-level semantic relationship between two sentences. We remark that our solution provides better interpretability of semantic textual similarity compared to the case of average pooling. For the sentence distance in Equation (3), TAVG assigns non-zero values to all token pairs that include irrelevant pairs; this makes it difficult to interpret the result. On the contrary, TRCMD 1 in Equation (4) is designed to explicitly involve the most relevant token pairs across sentences for the sentence distance, which allows us to interpret the result easily. We present a contrastive learning framework for RCMD (CLRCMD) that incorporates RCMD into the state-of-the-art contrastive learning framework. To this end, we convert RCMD to the corresponding similarity by s RCMD 1 M = 1  d RCMD 1 M : s RCMD 1 M ( s 1 , s 2 ) = 1 L 1 L 1 (cid:88) i =1 max j cos( x 1 i , x 2 j ) . s RCMD 2 M is computed in the same manner as well, and we average them to consider bidirectional semantic alignment between two sentences; this provides diverse gradient signals during optimization. The final similarity is described by s RCMD M ( s 1 , s 2 ) := 1 2 (cid:16) s RCMD 1 M ( s 1 , s 2 ) + s RCMD 2 M ( s 1 , s 2 ) (cid:17) . Adopting this similarity measure, the contrastive learning objective for the i -th sentence pair in a training batch is defined as follows:  log exp( s RCMD M ( s i , s i + ) / ) (cid:80) Bj =1 (exp( s RCMD M ( s i , s j + ) / ) + exp( s RCMD M ( s i , s j  ) / )) , where  is the temperature parameter and B is the batch size. Following (Gao et al., 2021), CLRCMD uses the other sentences in the batch to generate negative pairs. We argue that CLRCMD enhances both the sentence similarity and its interpretability in the following aspects. First, CLRCMD alleviates the catastrophic forgetting of pretrained semantics during 5972 Model STS12 STS13 STS14 STS15 STS16 STS-B SICK-R Avg BERT base -avg 29.12 59.96 47.22 60.61 63.72 47.20 58.25 52.30 SBERT base  70.97 76.53 73.19 79.09 74.30 77.03 72.91 74.86 SBERT base -flow  69.78 77.27 74.35 82.01 77.46 79.12 76.21 76.60 SBERT base -whitening  69.65 77.57 74.66 82.27 78.39 79.52 76.91 77.00 SimCSE cls -BERT base  75.30 84.67 80.19 85.40 80.82 84.25 80.39 81.57 SimCSE avg -BERT base 75.88 83.28 80.26 86.06 81.33 84.91 79.94 81.67 CLRCMD-BERT base 75.23 85.06 80.99 86.26 81.50 85.21 80.49 82.11 RoBERTa base -avg 32.50 55.78 45.00 60.61 61.68 55.31 61.66 53.22 SRoBERTa base  71.54 72.49 70.80 78.74 73.69 77.77 74.46 74.21 SRoBERTa base -whitening  70.46 77.07 74.46 81.64 76.43 79.49 76.65 76.60 SimCSE cls -RoBERTa base  76.53 85.21 80.95 86.03 82.57 85.83 80.50 82.52 SimCSE avg -RoBERTa base 75.75 85.10 80.85 85.95 83.33 85.55 79.41 82.28 CLRCMD-RoBERTa base 75.68 85.76 80.92 86.58 83.48 85.89 81.01 82.76 Table 1: The results on 7 STS benchmarks. the finetuning process. Its token-level supervision is produced by leveraging the textual semantics encoded in a pretrained checkpoint, because token pairs are semantically aligned according to their similarities in the contextualized embedding space. Namely, CLRCMD updates the parameters to improve the quality of sentence similarity while less breaking token-level semantics in the pretrained checkpoint. Furthermore, CLRCMD directly distills the relevance of a sentence pair into the relevance of semantically-aligned token pairs. In this sense, our contextualized embedding space effectively captures the token-level semantic relevance from training sentence pairs, which provides better interpretation for its sentence similarity. To analyze our approach in various viewpoints, we design and conduct experiments that focus on the following three research questions: RQ1 Does CLRCMD effectively measure sentence similarities using a pretrained language model? RQ2 Does CLRCMD provide the interpretation of sentence similarity which is well aligned with human judgements? RQ3 Does CLRCMD efficiently compute its sentence similarity for training and inference? We finetune a pretrained model using CLRCMD in the following settings. Following previous work (Gao et al., 2021), we use NLI datasets with hard negatives: SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). We use a pretrained backbone attached with a single head, which is the same with (Gao et al., 2021). As the initial checkpoint of the pretrained models, we employ bert-base-uncased and roberta-base provided by huggingface (Devlin et al., 2019; Liu et al., 2019). Adam optimizer is used with the initial learning rate 5 e  5 and linear decay schedule. Fp16 training is enabled where the maximum batch size is 128 on a single V100 GPU, and the softmax temperature is set to  = 0 . 05 (Gao et al., 2021). The training is proceeded with 4 different random seeds and the best model is chosen using the best Spearman correlation on STSb validation set which is evaluated every 250 steps during training. We evaluate the similarity model finetuned CLRCMD for STS task to quantitatively measure the quality of sentence similarity ( RQ1 ). Metric We measure Spearman correlation for each of seven STS benchmarks and calculate their average to compare the capability of representing sentences in general (Conneau and Kiela, 2018). Baselines We select the baselines that leverage a pretrained model, and they turn out to outperform other traditional baselines. We only list the baseline names for BERT base below; the names for RoBERTa base are obtained by replacing BERT base 5973 with RoBERTa base . BERT base -avg generates sentence embeddings by averaging the token embeddings from BERT base without finetuning. It indicates zero-shot performance of a checkpoint. SBERT base (Reimers and Gurevych, 2019) is a pioneering work to finetune a pretrained model for sentence embeddings. It trains a Siamese network using NLI datasets. SimCSE cls -BERT base (Gao et al., 2021) adopts a contrastive learning framework (Chen et al., 2020) using SNLI and MNLI datasets. The contextualized embedding of [CLS] is used as a sentence embedding. SimCSE avg -BERT base (Gao et al., 2021) is the same with SimCSE cls -BERT base except that it performs average pooling on token embeddings to obtain a sentence embedding. Result Table 1 reports Spearman correlation for each dataset and their average. For most of the datasets, CLRCMD shows higher correlation compared to the state-of-the-art baselines. In particular, for STS14, STS15, SICK-R datasets, CLRCMD-BERT base achieves comparable performance to SimCSE cls -RoBERTa base whose backbone language model is pretrained with 10 times larger data compared to BERT base . This implies that finetuning with token-level supervision from CLRCMD achieves the performance as good as using an expensively pretrained checkpoint. Next, we measure the performance of our approach on interpretable STS (iSTS) tasks in order to validate that CLRCMD embeds a sufficient level of interpretability even without any supervision (i.e., labeled training data) about semantically-aligned chunk pairs ( RQ2 ). Experimental setup We utilize the images and headlines data sources included in SemEval2016 Task 2: iSTS (Agirre et al., 2016). We measure the agreement between human judgement (gold semantic alignment across sentences) and the contributions of all token pairs to sentence similarity (element-wise multiplication of (1  M ) and T ). One challenge to use our similarity model for this task is to convert token pair contributions into chunk-level alignment. First, we summarize token pair contributions into chunk Model images headlines BERT base -avg 82.45 85.98 BERT base -RCMD 83.00 88.25 SimCSE avg -BERT base 82.98 85.80 CLRCMD-BERT base 87.25 90.55 RoBERTa base -avg 61.68 52.01 RoBERTa base -RCMD 82.44 88.92 SimCSE avg -RoBERTa base 73.66 77.30 CLRCMD-RoBERTa base 84.93 88.45 Table 2: The results on SemEval2016 task 2: iSTS. pair contributions by applying simple average pooling based on the chunk mapping represented by c ( i ) = { k | is_overlap ( c i , t k ) } , where c i is the i -th chunk and t k is the k -th token in a sentence. 1 Then, to obtain the alignment based on the pairwise chunk contributions, we design a criterion for selecting confident chunk pairs ( i, j ) as follows: C i,j = 1 | c 1 ( i ) || c 2 ( j ) | c 1 ( i ) (cid:88) k c 2 ( j ) (cid:88) l T k,l M k,l , a ( i, j ) = I [ j = argmax j (cid:48) C i,j (cid:48) ]  I [ i = argmax i (cid:48) C i (cid:48) ,j ] . Using the aligned chunk pairs obtained by each method, we compute the alignment F1 score as the evaluation metric, which indicates the agreement between human judgement and chunk contribution. 2 We consider eight different configurations to investigate the effectiveness of the following components: 1) sentence similarity, 2) contrastive learning, and 3) pretrained checkpoints. Result Table 2 shows the clear tendency of iSTS performance with respect to each of the above components. First of all, the token pair contribution from RCMD is more consistent with human judgement than that from average pooling. RCMD improves alignment F1 scores even without finetuning (BERT base -RCMD and RoBERTa base -RCMD), indicating that RCMD effectively discovers the token-level relevance encoded inside a pretrained checkpoint. In addition, the alignment F1 score increases when we finetune a model using CLRCMD. Notably, CLRCMD-BERT base successfully improves the alignment F1 score whereas 1 We use gold standard chunking information to focus on alignment only, which is the second subtrack in iSTS. SimCSE avg -BERT base does not. This result shows that finetuning a model using the similarity measure based on semantically-aligned token pairs (i.e., fine-grained supervision induced by RCMD) further enhances the interpretability of a model. We qualitatively analyze the sentence similarity from the perspective of the transportation problem in order to demonstrate that a model trained by CLRCMD provides clear and accurate explanation ( RQ2 ). To this end, we visualize the contribution of token pairs obtained from CLRCMD-BERT base and that from SimCSE avg -BERT base , and then clarify how their sentence similarity is computed differently from each other. Three sentence pairs are randomly selected from STS13 dataset. Figure 2 illustrates the token pair contribution heatmap for positive, neutral, and negative sentence pairs. CLRCMD vs. SimCSE avg Overall, CLRCMD aligns two sentences better than the baseline. To be specific, CLRCMD effectively highlights the contributions of semantically-relevant token pairs and excludes the other contributions (Figure 2 up-per). On the contrary, SimCSE avg fails to represent meaningful token-level relevance for sentence similarity (Figure 2 lower). The rank-1 constraint of SimCSE avg prevents the model from getting any plausible alignment between two sentences, while it simply tunes the contributions of all possible token pairs at once. We emphasize that the super-fluous correlation in the heatmap not only inhibits the capability to capture sentence similarity, but also makes it difficult for humans to understand how sentence similarity is computed. Case study on positive, neutral, and negative sentence pairs For the positive pair (Figure 2 left), CLRCMD clearly matches all semantically-aligned token pairs including the linking words ({,, and}{and}), synonyms ({realize, comprehend}{comprehend}), and omitted contexts ({the}{the nature of, of}). For the neutral pair (Figure 2 middle), the two sentences have the same lexical structure except for the date. In this case, CLRCMD assigns low contributions to the token pairs about day and month ({25, august}{19, july}), while keeping the contributions high for all the other pairs of identical tokens. Therefore, end-users can clearly figure out which part is semantically different based on their contributions as well as alignment. In case of the negative pair (Figure 2 right), both the models are not able to find any plausible alignment; CLRCMD lowers contributions for most of the token pairs ex-5975 Batch size 16 32 64 128 RCMD dense 7.5 22.2 OOM OOM RCMD sparse 4.6 6.1 10.6 25.8 Table 3: GPU memory usage (GB) of CLRCMD with various batch sizes. cept the token pair with identical contents (after riots). That is, end-users also can interpret the negative pair based on the heatmap where semantic correspondence between two sentences does not clearly exist but few overlapped tokens highly contribute to the sentence similarity. We measure GPU memory usage and inference time of CLRCMD-BERT base to demonstrate that CLRCMD can be executed on a single GPU and an inference of our model takes almost the same cost to that of the baseline ( RQ3 ). Implementation of RCMD We implement two variants of RCMD, RCMD dense and RCMD sparse , to investigate the effect of exploiting the sparseness in RCMD. Both of them calculate sentence distance by the sum of element-wise multiplications of the cost matrix and the transportation matrix. For an input sentence pair, RCMD dense maintains the full pairwise token distances ( MCMD ), whereas RCMD sparse only keeps the token distances at which the transportation matrix has nonzero values ( { MCMD i,j | TCMD i,j (cid:54) = 0 } ). Note that the number of nonzero entries in the transportation matrix of RCMD is at most 2 L , which is an order of magnitude smaller than the number of all entries, L 2 . Result Table 3 reports the GPU memory usage during the finetuning process. For batch-wise contrastive learning, GPU memory requirement becomes O ( B 2 ) in terms of the batch size B , because all pairwise sentence similarities within a batch need to be computed. In this situation, RCMD dense using a dense matrix drastically increases GPU memory usage by O ( B 2 L 2 ) , and as a result, the batch size cannot grow larger than 32. In contrast, RCMD sparse successfully enlarges the batch size up to 128 by exploiting sparseness in the transportation matrix of RCMD, which eventually reduces the space complexity to O ( B 2 L ) . Experimental setup We measure the time for predicting the similarities of 512 sentence pairs on a single V100 GPU while increasing the sequence length from 8 to 128, which is the most influential factor for inference time. We repeat this process 10 times and report the average values. Result Figure 3 shows the average elapsed time for inference. The model with RCMD takes almost the same inference time as the model with the simple average pooling-based similarity. We highlight that 98% of the sentences in STS13 dataset consist of at most 48 tokens and particularly, the time difference is negligible in case of predicting the sentence pairs whose sentences have less than 48 tokens. This result shows that significant increment of inference time does not occur within the range of the sequence length owing to parallel GPU computations, even though RCMD has the quadratic time complexity with respect to the sentence length. In this work, we present CLRCMD, a learning framework for an interpretable sentence similarity model based on optimal transport. First, we view each sentence similarity measure as a transportation problem, pointing out the unexpressiveness of the existing pooling-based similarity. Integrating the concept of optimal transport into a pretrained language model, CLRCMD defines the distance measure by using the semantically-aligned token pairs between two sentences and furthermore, it finetunes a model with this distance based on contrastive learning for better interpretability. We empirically show that CLRCMD accurately predicts 5976 sentence similarity while providing interpretable token pair contributions consistent with human judgements. With the belief that the ability to interpret model behavior is critical for future AI models, we focus on enhancing this virtue targeted on STS task throughout this research. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2018-0-00584, (SW starlab) Development of Decision Support System Software based on Next-Generation Machine Learning) and the NRF grant funded by the MSIT (South Korea, No.2020R1A2B5B03097210) ) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01906, Artificial Intelligence Graduate School Pro-gram(POSTECH)).
abstain abstain method objective objective other abstain abstain result abstain abstain abstain abstain abstain objective objective objective method abstain objective objective abstain objective abstain method method abstain result other other other other other other other other other method other other other other other other other method method method abstain abstain abstain method other abstain abstain abstain abstain method abstain abstain abstain other other abstain abstain abstain method method method method method abstain method abstain abstain abstain abstain abstain method abstain method method method method abstain other abstain method other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method objective abstain objective method other