Selecting input features of top relevance has become a popular method for building self-explaining models. In this work, we extend this selective rationalization approach to text matching, where the goal is to jointly select and align text pieces, such as tokens or sentences, as a justification for the downstream prediction. Our approach employs optimal transport (OT) to find a minimal cost alignment between the inputs. However, directly applying OT often produces dense and therefore uninterpretable alignments. To overcome this limitation, we introduce novel constrained variants of the OT problem that result in highly sparse alignments with controllable sparsity. Our model is end-to-end differentiable using the Sinkhorn algorithm for OT and can be trained without any alignment annotations. We evaluate our model on the StackExchange, MultiNews, e-SNLI, and MultiRC datasets. Our model achieves very sparse rationale selections with high fidelity while preserving prediction accuracy compared to strong attention baseline models. 1 Introduction The growing complexity of deep neural networks has given rise to the desire for self-explaining models (Li et al., 2016; Ribeiro et al., 2016; Zhang et al., 2016; Ross et al., 2017; Sundararajan et al., 2017; Alvarez-Melis and Jaakkola, 2018b; Chen et al., 2018a). In text classification, for instance, one popular method is to design models that can perform classification using only a rationale , which is a subset of the text selected from the model input that fully explains the model's prediction (Lei et al., 2016; Bastings et al., 2019; Chang et al., 2019). This selective rationalization method, often * Denotes equal contribution. trained to choose a small yet sufficient number of text spans, makes it easy to interpret the model's prediction by examining the selected text. In contrast to classification, very little progress has been made toward rationalization for text matching models. The task of text matching encompasses a wide range of downstream applications, such as similar document recommendation (dos Santos et al., 2015), question answering (Lee et al., 2019), and fact checking (Thorne et al., 2018). Many of these applications can benefit from selecting and comparing information present in the provided documents. For instance, consider a similar post suggestion in a tech support forum as shown in Figure 1. The extracted rationales could provide deeper insights for forum users while also helping human experts validate and improve the model. In this work, we extend selective rationalization for text matching and focus on two new challenges that are not addressed in previous rationalization work. First, since text matching is fundamentally about comparing two text documents, rationale selection should be jointly modeled and optimized for matching. Second, the method should produce an interpretable alignment between the selected rationales showcasing their relations for the downstream prediction. This is very different from rationalization for text classification, where the selection is performed independently on each input text and an alignment between rationales is unnecessary. One popular method for aligning inputs is attention-based models (Bahdanau et al., 2015; Rocktschel et al., 2015; Rush et al., 2015; Xu et al., 2015; Kim et al., 2018). However, a limitation of neural attention is that the alignment is rarely sparse, thus making it difficult to interpret how the numerous relations among the text spans lead to the model's prediction. Recent work has explored sparse variants of attention (Martins and Astudillo, 2016; Niculae and Blondel, 2017; Lin et al., 2018; Malaviya et al., 2018; Niculae et al., 2018), but the number of non-zero alignments can still be large (Laha et al., 2018). Additionally, because of the heavy non-linearity following most attention layers, it is difficult to truly attribute the model's predictions to the alignment, which means that attention-based models lack fidelity. We propose to address these challenges by directly learning sparse yet sufficient alignments using optimal transport (OT). We use OT as a building block within neural networks for determining the alignment, providing a deeper mathematical justification for the rationale selection. In order to produce more interpretable rationales, we construct novel variants of OT that have provable and controllable bounds on the sparsity of the alignments. Selecting and aligning text spans can be jointly optimized within this framework, resulting in optimal text matchings. Our model is fully end-to-end differentiable using the Sinkhorn algorithm (Cu-turi, 2013) for OT and can be used with any neural network architecture. We evaluate our proposed methods on the StackExchange, MultiNews (Fabbri et al., 2019), e-SNLI (Camburu et al., 2018), and MultiRC (Khashabi et al., 2018) datasets, with tasks ranging from similar document identification to reading comprehension. Compared to other neural baselines, our methods show comparable task performance while selecting only a fraction of the number of alignments. We further illustrate the effectiveness of our method by analyzing how faithful the model's predictions are to the selected rationales and by comparing the rationales to human-selected rationales provided by DeYoung et al. (2019) on the e-SNLI and MultiRC datasets. Selective Rationalization. Model interpretability via selective rationalization has attracted considerable interest recently (Lei et al., 2016; Li et al., 2016; Chen et al., 2018a; Chang et al., 2019). Some recent work has focused on overcoming the challenge of learning in the selective rationalization regime, such as by enabling end-to-end differentiable training (Bastings et al., 2019) or by regularizing to avoid performance degeneration (Yu et al., 2019). Unlike these methods, which perform independent rationale selection on each input document, we extend selective rationalization by jointly learning selection and alignment, as it is better suited for text matching applications. Concurrent to this work, DeYoung et al. (2019) introduce the ERASER benchmark datasets with human-annotated rationales along with several rationalization models. Similarly to DeYoung et al. (2019), we measure the faithfulness of selected rationales, but our work differs in that we additionally emphasize sparsity as a necessary criterion for interpretable alignments. Alignment. Models can be made more interpretable by requiring that they explicitly align related elements of the input representation. In NLP, this is often achieved via neural attention (Bah-danau et al., 2015; Chen et al., 2015; Rush et al., 2015; Cheng et al., 2016; Parikh et al., 2016; Xie et al., 2017). Many variants of attention, such as temperature-controlled attention (Lin et al., 2018) and sparsemax (Martins and Astudillo, 2016), have been proposed to increase sparsity within the attention weights. However, it is still debatable whether attention scores are truly explanations (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Distance-based methods of aligning text have also been proposed (Li et al., 2019), but they similarly cannot guarantee sparsity or explainability. In this work, we explicitly optimize rationale selection and alignment as an integral part of the model and evaluate the degree to which the alignment explains the model's predictions. Optimal Transport. The field of optimal transport (OT) began with Monge (1781), who explored the problem of determining a minimal cost assignment between sets of equal sizes. Kantorovich (1942) relaxed Monge's problem to that of determining an optimal transport plan for moving probability mass between two probability distributions. Since the introduction of a differentiable OT solver by Cuturi (2013), OT has seen many applications in deep learning and NLP, such as topic embedding (Kusner et al., 2015), text generation (Chen et al., 2018b), cross-lingual word embedding alignment (Alvarez-Melis and Jaakkola, 2018a), graph embedding (Xu et al., 2019), and learning permutations (Mena et al., 2018). Peyr and Cuturi (2019) provides an overview of the computational aspects of OT. Unlike prior work, we develop novel additional constraints on the OT problem that produce particularly sparse and interpretable alignments. Consider two related text documents D x and D y . These documents are broken down into two sets of text spans, S x and S y , where the text spans can be words, sentences, paragraphs, or any other chunking of text. These text spans are then mapped to vector representations using a function g (  ) (e.g., a neural network), which produces two sets of vectors representing the inputs, X = { x i } ni =1 = { g ( S xi ) } ni =1 and Y = { y i } mi =1 = { g ( S yi ) } mi =1 , where x i , y i  R d . We define an interpretable text matching as an alignment between the text spans in X and Y that explains the downstream prediction. Following common practice for previous self-explaining models (Lei et al., 2016; Alvarez-Melis and Jaakkola, 2018b), we specify that a desirable model must produce alignments satisfying the following criteria of interpretability. Explicitness. The alignment between text spans generated by the model should be an observable and understandable component of the model. Our model explicitly encodes the alignment between X and Y as a matrix P  R n  m + where P i,j indicates the degree to which x i and y j are aligned. Sparsity. In order for the alignment to be interpretable, the alignment matrix P must be sparse, meaning there are very few non-zero alignments between the text spans. A sparser alignment is easier to interpret as fewer alignments between text spans need to be examined. Faithfulness. An interpretable text matching is only meaningful if the model's predictions are faithful to the alignment, meaning the predictions are directly dependent on it. Similarly to previous work, our model achieves faithfulness by using only the selected text spans (and their representations) for prediction. That is, the selected rationales and alignment should be sufficient to make accurate predictions. In addition to sufficiency, faithfulness also requires that the model output should be easily attributed to the choice of alignment 1 . For simple attribution, we define our model output as either a linear function of the alignment P or a shallow feed-forward network on top of P . In the following sections, we introduce optimal transport as a method to produce interpretable text matchings satisfying all three desiderata. An instance of the discrete optimal transport problem consists of two point sets, X = { x i } ni =1 and Y = { y i } mi =1 , with x i , y i  R d . Additionally, X and Y are associated with probability distributions a   n and b   m , respectively, where  n is the probability simplex  n := (cid:8) p  R n + : (cid:80) ni =1 p i = 1 (cid:9) . A cost function c ( x , y ) : R d  R d  R specifies the cost of aligning a pair of points x and y . The costs of aligning all pairs of points are summarized by the cost matrix C  R n  m , where C i,j = c ( x i , y j ) . The goal of optimal transport is to compute a mapping that moves probability mass from the points of X (distributed according to a ) to the points of Y (distributed according to b ) so that the total cost of moving the mass between points is minimized according to the cost function c . This mapping is represented by a transport plan, or alignment matrix, P  R n  m + , where P i,j indicates the amount of probability mass moved from x i to y j . The space of valid alignment matrices is the set U ( a , b ) := { P  R n  m + : P 1 m = a , PT 1 n = b } since P must marginalize out to the corresponding probability distributions a and b over X and Y . Under this formulation, the optimal transport problem is to find the alignment matrix P that minimizes the sum of costs weighted by the alignments: LC ( a , b ) := min P  U ( a , b ) (cid:104) C , P (cid:105) = (cid:88) i,j C i,j P i,j . Note that this optimization is a linear programming problem over the convex set U ( a , b ) . As a result, one of the extreme points of U ( a , b ) must be an optimal solution. 1 For example, a linear model achieves strong attribution because the importance of each input feature is a constant parameter. Optimal transport is known to produce alignments that are especially sparse. In particular, the following propositions characterize the extreme point solution P  of LC ( a , b ) and will be important in designing interpretable alignments in Section 5. Proposition 1 (Brualdi (2006), Thm. 8.1.2) . Any extreme point P  that solves LC ( a , b ) has at most n + m  1 non-zero entries. Proposition 2 (Birkhoff (1946)) . If n = m and a = b = 1 n /n , then every extreme point of U ( a , b ) is a permutation matrix. In other words, while the total number of possible aligned pairs is n  m , the optimal alignment P  has O ( n + m ) non-zero entries. Furthermore, if n = m , then any extreme point solution P  is a permutation matrix and thus only has O ( n ) nonzero entries. Figure 2 illustrates two alignments, including one that is a permutation matrix. Note that the optimal solution of LC ( a , b ) may not be unique in degenerate cases, such as when C i,j is the same for all i, j . In such degenerate cases, any convex combination of optimal extreme points is a solution. However, it is possible to modify any OT solver to guarantee that it finds an extreme point (i.e., sparse) solution. We provide a proof in Appendix D, although experimentally we find that these modifications are unnecessary as we nearly always obtain an extreme point solution. LC ( a , b ) is a linear programming problem and can be solved exactly with interior point methods. Recently, Cuturi (2013) proposed an entropy-regularized objective that can be solved using a fully differentiable, iterative algorithm, making it ideal for deep learning applications. Specifically, the entropy-regularized objective is L (cid:15) C ( a , b ) := min P  U ( a , b ) (cid:104) C , P (cid:105)  (cid:15) H ( P ) , where H ( P ) is the entropy of alignment matrix P and (cid:15) > 0 controls the amount of entropy regularization. In practice, (cid:15) can be set sufficiently small such that the solution to L (cid:15) C ( a , b ) is a good approximation of the solution to LC ( a , b ) . Conveniently, L (cid:15) C ( a , b ) has a solution of the form P  = diag( u ) K diag( v ) , where K = e  C /(cid:15) and ( u , v )  R n +  R m + . The vectors u and v can be determined using the Sinkhorn-Knopp matrix scaling algorithm (Sinkhorn and Knopp, 1967), (a) Alignment 1 (graph) (b) Alignment 2 (graph) (c) Alignment 1 (matrix) Since each iteration consists only of matrix operations, the Sinkhorn algorithm can be used as a differentiable building block in deep learning models. For instance, in this work we take C as the distance between hidden representations given by a parameterized neural network encoder. Our model performs the Sinkhorn iterations until convergence (or a maximum number of steps) and then outputs the alignment P and the total cost (cid:104) C , P (cid:105) as inputs to subsequent components of the model. Using vanilla OT produces sparse alignments as guaranteed by Proposition 1, but the level of sparsity is insufficient to be interpretable. For instance, Alignment 1 in Figure 2 still has a significant number of non-zero alignment values. Motivated by this limitation, we propose to encourage greater sparsity and interpretability by constructing OT problems with additional constraints. General Recipe for Additional Constraints. Intuitively, an interpretable alignment should be sparse in two ways. First, each text span should be aligned to one or a very small number of spans in the other input text. Second, the total number of Figure 3: An illustration of the process of computing a one-to-two assignment between the points of X and Y . aligned pairs should be small enough so that the alignment can be easily examined by a human. We modify the OT problem in several ways to guarantee both aspects of sparsity. We start by forcing the solution to be an assignment , which is a one-to-one (or one-to-few) alignment such that every non-zero entry in the alignment matrix is equal, thereby simplifying interpretability. Alignment 2 in Figure 2 is an example of a one-to-one assignment. We also consider two other constructions, one that makes every text span in the alignment optional and another that directly limits the total number of aligned pairs. Replica points are exact copies of the original points in X or Y and can be used to control the sparsity of each point's alignment. Dummy points , also known as tariff-free reservoirs in prior work, are points that can be aligned to with 0 cost. Dummy points are used for absorbing unused probability mass in partial transport, where the constraints are relaxed to P 1 m  a and PT 1 n  b (Caffarelli and McCann, 2010; Figalli, 2010). The idea is to add an appropriate number of replica points and dummy points to create  X and  Y with |  X | = |  Y | = N for some N . Then by using uniform probability distributions a = b = 1 N /N , Proposition 2 implies that one of the solutions to the OT problem will be a permutation matrix, i.e., a one-to-one assignment between the points in  X and  Y . Since the points of X and Y are included in  X and  Y , we can directly extract an assignment between X and Y from the assignment between  X and  Y . Figure 3 illustrates the procedure. Note that the same solution can be attained without explicitly replicating any points by adjusting the probability distributions a and b , but we use replication for ease of exposition. Also note that the Sinkhorn algorithm is compatible with replica and dummy points and the model remains differentiable. We now describe three specific instances of this procedure that produce interpretable assignments with different sparsity patterns. Without loss of generality, we assume that n = | X |  | Y | = m . One-tok Assignment. In this assignment, every point in the smaller set X should map to k points in the larger set Y , where k  { 1 , 2 , . . . , (cid:98) mn (cid:99)} . This will result in a sparsity of kn  (cid:98) mn (cid:99) n  m . To compute such an assignment, we set  Y = Y and we construct  X with k copies of every point in X along with m  kn dummy points. Since |  X | = |  Y | = m , applying OT to  X and  Y produces a one-to-one assignment between  X and  Y . As  X contains k replicas of each point in X , each unique point in X is mapped to k points in Y , thus producing a one-tok assignment. The remaining m  kn mappings to dummy points are ignored. Relaxed One-tok Assignment. In a relaxed one-tok assignment, each point in X can map to at most k points in Y . As with the one-tok assignment, we use k replicas of each point in X , but now we add m dummy points to X and kn dummy points to Y , meaning |  X | = |  Y | = m + kn . Because of the number of replicas, this will produce at most a one-tok assignment between X and Y . However, since there is now one dummy point in  Y for every original point in  X , every original point has the option of aligning to a dummy point, resulting in at most k alignments. Note that in this case, the cost function must take both positive and negative values to prevent all original points from Constraint #R of X #D in X (cid:48) #D in Y (cid:48) Sparsity ( s ) Vanilla 1 0 0 s  n + m  1 One-tok k m  kn 0 s = kn  m R one-tok k m kn s  kn  m Exactk 1 m  k n  k s = k  n Table 1: Summary of constrained alignment construction and sparsity. Exactk Assignment. An exactk assignment maps exactly k points in X to points in Y , where k  n . An exactk assignment can be constructed by adding m  k dummy points to X and n  k dummy points to Y , meaning |  X | = |  Y | = n + m  k . In this case, the cost function must be strictly positive so that original points map to dummy points whenever possible. This leaves exactly k alignments between original points in X and Y . Controllable Sparsity. Table 1 summarizes the differences between vanilla OT and the constrained variants. The freedom to select the type of constraint and the value of k gives fine-grained control over the level of sparsity. We evaluate the performance of all these variants in our experiments. Datasets. We evaluate our model and all baselines on four benchmarks: two document similarity tasks, MultiNews and StackExchange, and two classification tasks, e-SNLI and MultiRC. The e-SNLI and MultiRC tasks come from the ERASER benchmark (DeYoung et al., 2019), which was created to evaluate selective rationalization models. We chose those two datasets as they are best suited for our text matching setup. StackExchange 2 is an online question answering platform and has been used as a benchmark in previous work (dos Santos et al., 2015; Shah et al., 2018; Perkins and Yang, 2019). We took the June 2019 data dumps 3 of the AskUbuntu and SuperUser subdomains of the platform and combined them to form our dataset. MultiNews (Fabbri et al., 2019) is a multi-document summarization dataset where 2 to 10 news articles share a single summary. We consider 2 https://stackexchange.com/sites 3 https://archive.org/details/ stackexchange Metric StackExchange MultiNews # docs 730,818 10,130 # similar doc pairs 187,377 22,623 Avg sents per doc 3.7 31 Max sents per doc 54 1,632 Avg words per doc 87 680 Vocab size 603,801 299,732 Table 2: Statistics for the document ranking datasets. every pair of articles that share a summary to be a similar document pair. Table 2 shows summary statistics of the two document ranking datasets. e-SNLI (Camburu et al., 2018) is an extended version of the SNLI dataset (Bowman et al., 2015) for natural language inference where the goal is to predict the textual entailment relation (entailment, neutral, or contradiction) between premise and hypothesis sentences. Human rationales are provided as highlighted words in the two sentences. MultiRC (Khashabi et al., 2018) is a reading comprehension dataset with the goal of assigning a label of true or false to a question-answer pair depending on information from a multi-sentence document. We treat the concatenated question and answer as one input and the document as the other input for text matching. Human rationales are provided as highlighted sentences in the document. For StackExchange and MultiNews, we split the documents into 80% train, 10% validation, and 10% test, while for e-SNLI and MultiRC, we use the splits from DeYoung et al. (2019). Metrics. We evaluate models according to the following three criteria. 1. Sparsity. To evaluate sparsity, we compute the average percentage of active alignments produced by each model, where an alignment is active if it exceeds a small threshold  . This threshold is necessary to account for numerical imprecision in alignment values that are essentially zero. We set  = 0 . 01 n  m unless otherwise specified, where n and m are the number of text spans in the two documents. 2. Sufficiency. If a model makes a correct prediction given only the rationales, then the rationales are sufficient. We evaluate sufficiency by providing the model only with active alignments and the aligned text representations and by masking non-active inputs (using the threshold  ). 3. Relevance. The relevance of rationales is determined by whether a human would deem them valid and relevant. We compute relevance using the token-level F1 scores of model-generated rationales compared to human-selected rationales on the e-SNLI and MultiRC datasets. We also perform a qualitative human evaluation. Baselines and Implementation Details. We use the decomposable attention model (Parikh et al., 2016) as our baseline attention model. In addition, we compare our model to two attention variants that are designed to encourage sparsity. The temperature attention variant applies a temperature term T in the softmax operator (Lin et al., 2018). The sparse attention variant adopts the sparsemax operator (Martins and Astudillo, 2016) in place of softmax to produce sparse attention masks. Our constrained OT model operates as illustrated in Figure 4. After splitting the input documents into sentences, our model independently encodes each sentence and computes pairwise costs between the encoded representations 4 . Dummy and replica encodings are added as needed for the desired type of constrained alignment. Our model then applies OT via the Sinkhorn algorithm to the cost matrix C to produce an optimal alignment matrix P . For the document ranking tasks, the final score is simply (cid:104) C , P (cid:105) . For the classification tasks, we use the alignment P as a sparse mask to select encoded text representations, and we feed the aggregated representation to a shallow network to predict the output label, similar to our baseline attention models. For a fair comparison, our models and all baselines use the same neural encoder to encode text spans before the attention or OT operation is applied. Specifically, we use RoBERTa (Liu et al., 2019), a state-of-the-art pre-trained encoder, for 4 For the e-SNLI dataset, where documents are single sentences, we use the contextualized token representations from the output of the sentence encoder following previous work (Thorne et al., 2019). the StackExchange and MultiRC dataset. We use use bi-directional recurrent encoders (Lei et al., 2018) for the MultiNews and e-SNLI datasets 5 . The value of k for the OT constraints is chosen for each dataset by visually inspecting alignments in the validation set, though model performance is robust to the choice of k . In order to compare our models' rationales to human annotations, we use a binary thresholding procedure as described in Appendix C. We report results averaged over 3 independent runs for each model. Additional implementation details are provided in Appendix C. 7 Results Synthetic Visualizations. Before experimenting with the datasets, we first analyze the alignments obtained by different methods on a synthetic cost matrix in Figure 5. As shown in the figure, all attention baselines struggle to produce sufficiently sparse alignments, even with the use of a small temperature or the sparsemax operator. In contrast, our methods are very sparse, as a result of the provable sparsity guarantees of the constrained alignment 5 The input text in the MultiNews dataset is too long for large BERT models. The e-SNLI dataset in ERASER contains human-annotated rationales at the word level while BERT models use sub-word tokenization. problem. For instance, the relaxed one-tok assignment produces fewer active alignments than either the number of rows or columns, and the exactk assignment finds exactly k = 4 alignments. StackExchange & MultiNews. Table 3 presents the results of all models on the StackExchange and MultiNews datasets. We report standard ranking and retrieval metrics including area under the curve (AUC), mean average precision (MAP), mean reciprocal rank (MRR), and precision at 1 (P@1). The results highlight the ability of our methods to obtain high interpretability while retaining ranking performance comparable to strong attention baselines. For example, our model is able to use only 6 aligned pairs to achieve a P@1 of 96.6 on the MultiNews dataset. In comparison, the sparse attention model obtains a P@1 of 97.1 but uses more than 300 alignment pairs and is thus difficult to interpret. Model complexity and speed on the StackExchange dataset are reported in Table 7 in Appendix C. e-SNLI. Table 4 shows model performance on the e-SNLI dataset. As with document similarity ranking, we evaluate classification accuracy when the model uses only the active alignments. This is to ensure faithfulness, meaning the model truly and exclusively uses the rationales to make predictions. Since attention is not explicitly trained to use only active alignments, we also report the accuracy of attention models when using all attention weights. As shown in the table, the accuracy of attention methods decreases significantly when we remove attention weights other than those deemed active by the threshold  . In contrast, our model retains high accuracy even with just the active alignments since sparsity is naturally modeled in our contrained optimal transport framework. Figure 6 visualizes the Figure 6: Model accuracy on the e-SNLI dataset when using different percentages of tokens as rationales. change to model accuracy when different proportions of tokens are selected by the models. Table 4 also presents the token-level F1 scores for the models' selected rationales compared to human-annotated rationales. Note that the rationale annotations for this task are designed for token selection rather than alignment and are sometimes only on one of the input sentences. Nevertheless, our model obtains F1 scores on par with recent work (DeYoung et al., 2019; Thorne et al., 2019). MultiRC. Table 5 presents the results on the MultiRC dataset. Compared to attention models, our OT-based models achieve similar task performance with a higher rationale F1 score, despite selecting fewer rationales. The model variants from DeYoung et al. (2019) in general achieve higher task F1 performance. However, their unsupervised model suffers from degeneration due to the challenges of end-to-end training without rationale supervision. We also create supervised versions of our models that learn from the human-annotated rationales Model Accuracy Task F1 % Token Premise F1 Hypothesis F1 P&H F1 OT (relaxed 1:1) 82.4 82.4 69.1 25.1 43.7 34.6 OT (exact k = 4 ) 81.4 81.4 38.7 24.3 45.0 35.4 OT (exact k = 3 ) 81.3 81.4 29.6 28.6 50.0 39.8 OT (exact k = 2 ) 81.3 81.3 21.6 24.8 30.6 27.8 Attention 76.3 (82.1) 76.2 37.9 26.6 37.6 32.2 Attention ( T = 0 . 1 ) 73.9 (81.5) 73.9 33.0 28.4 44.1 36.5 Attention ( T = 0 . 01 ) 70.2 (81.4) 69.9 30.6 26.1 38.0 32.2 Sparse Attention 63.5 (75.0) 63.1 12.5 8.8 24.5 17.2 Thorne et al. (2019) (81.0) -22.2 57.8  Lei et al. (2016) -90.3 --37.9  Lei et al. (2016) (+S) -91.7 --69.2  Bert-To-Bert (+S) -73.3 --70.1 Table 4: e-SNLI accuracy, macro-averaged task F1, percentage of tokens in active alignments, and token-level F1 of the model-selected rationales compared to human-annotated rationales for the premise, hypothesis, and both (P&H F1). during training. These supervised models achieve comparable task performance to and better rationale F1 scores than models from DeYoung et al. (2019), demonstrating the strength of a sparse rationale alignment. Supervised training details can be found in Appendix C. Qualitative Studies. We performed a human evaluation on documents from StackExchange that reveals that our model's alignments are preferred to attention. The results of the human evaluation, along with examples of StackExchange and e-SNLI alignments, are provided in Appendix A. 8 Conclusion Balancing performance and interpretability in deep learning models has become an increasingly important aspect of model design. In this work, we propose jointly learning interpretable alignments as part of the downstream prediction to reveal how neural network models operate for text matching applications. Our method extends vanilla optimal transport by adding various constraints that produce alignments with highly controllable sparsity patterns, making them particularly interpretable. Our models show superiority by selecting very few alignments while achieving text matching performance on par with alternative methods. As an added benefit, our method is very general in nature and can be used as a differentiable hard-alignment module in larger NLP models that compare two pieces of text, such as sequence-to-sequence models. Furthermore, our method is agnostic to the underlying nature of the two objects being aligned and can therefore align disparate objects such as images and captions, enabling a wide range of future applications within NLP and beyond. We thank Jesse Michel, Derek Chen, Yi Yang, and the anonymous reviewers for their valuable discussions. We thank Sam Altschul, Derek Chen, Amit Ganatra, Alex Lin, James Mullenbach, Jen Seale, Siddharth Varia, and Lei Xu for providing the human evaluation.
abstain objective result abstain objective method method result abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain objective method objective abstain method abstain result method other other other objective abstain abstain other other other other other other method other other other other other objective abstain abstain other method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method other other other other other other other other other other other other other other other other other other other other other other other abstain other other other other other other other other other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain method abstain method abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain method abstain other abstain abstain abstain abstain abstain method abstain method abstain method abstain method abstain method abstain abstain abstain method abstain abstain method method abstain method method abstain abstain method abstain method abstain method method method abstain method abstain method abstain method method abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain method method method abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain method abstain method method method method method other other