Abstractive summarization, the task of generating a concise summary of input documents, requires: (1) reasoning over the source document to determine the salient pieces of information scattered across the long document, and (2) composing a cohesive text by reconstructing these salient facts into a shorter summary that faithfully reflects the complex relations connecting these facts. In this paper, we adapt TP-TRANSFORMER (Schlag et al., 2019), an architecture that enriches the original Transformer (Vaswani et al., 2017) with the explicitly compositional Tensor Product Representation (TPR), for the task of abstractive summarization. The key feature of our model is a structural bias that we introduce by encoding two separate representations for each token to represent the syntactic structure (with role vectors ) and semantic content (with filler vectors ) separately. The model then binds the role and filler vectors into the TPR as the layer output. We argue that the structured intermediate representations enable the model to take better control of the contents (salient facts) and structures (the syntax that connects the facts) when generating the summary. Empirically, we show that our TP-TRANSFORMER outperforms the Transformer and the original TPTRANSFORMER significantly on several abstractive summarization datasets based on both automatic and human evaluations. On several syntactic and semantic probing tasks, we demonstrate the emergent structural information in the role vectors and improved syntactic interpretability in the TPR layer outputs. 1 1 Introduction Abstractive summarization is the task of generating a shorter version of a source text without necessarily reusing the sentences from the original source,  Work partially done while at Microsoft Research. Original Text (Truncated): Authorities said the incident took place on Sao Joao beach in Caparica, south-west of Lisbon . The National Maritime Authority said a middle-aged man and a young girl died after they were unable to avoid the plane. [....] Other reports said the victims had been sunbathing when the plane made its emergency landing . [  ] Video footage from the scene carried by local broadcasters showed a small recreational plane parked on the sand, apparently intact and surrounded by beachgoers and emergency workers. [  ] Reference Summary: A man and a child have been killed after a light aircraft made an emergency landing on a beach in Portugal. while preserving the meaning of its salient contents. It is a complex task that requires: semantic understanding of the source text and reasoning over its lexical units, making inferences about their relation to extract salient facts which are scattered across the long document, as well as generating a concise and coherent sequence of new sentences that covers the salient facts. While humans are remarkably good at this type of reasoning and abstraction, developing models that are capable of extraction, comprehension, abstraction, and reformulation of salient contents has been an open research question. One prominent aspect of abstractive summarization is that models struggle with combining multiple salient aspects in the source text into a coherent and grammatical set of sentences that preserve the original information in the source document. As shown in Fig. 1, these pieces of salient information (death", emergency landing", beach") are often connected by complex syntactic, causal, and temporal relations and are loosely grouped under the main topic of the source document. The transformer models (Vaswani et al., 2017) encode syntactic and semantic information of the input text into a single representation space with the self-attention, and decode the salient aspects into a short summary with the cross-attention. However, despite the large number of training examples, current state-of-the-art transformer based approaches still struggle with systematic generalization of the composition of multiple salient pieces of information. In this paper, we investigate new types of computational primitives for transformers based on Tensor Product Representations (TPRs) (Smolen-sky, 1990) which are explicitly-compositional vector embeddings of symbolic structures. A Tensor Product Representation encodes a constituent in a symbolic structure as a composite of a role , which encodes the structural information (e.g., the dependency relation with another word), and a filler , which encodes the content of the constituent (e.g., the meaning of a word). Analogously, the TP-TRANSFORMER constructs a pair of representations for every token at every layer: a filler vector returned by attention and a novel role vector . As visualized in Fig. 2, the model then binds the role and filler vectors to produce the output of every token as a TPR. We adapt the TP-TRANSFORMER (Schlag et al., 2019), which was proposed for solving mathematics problems, for the task of abstractive summarization. Unlike the original TP-TRANSFORMER , which directly projects the input representation into a continuous role vector space, our model generates the role vectors by attending to a learned dictionary of role embeddings (Palangi et al., 2018). We observe that most learned role attention distributions are approximately one-hot, thus restricting the role vectors to a highly discrete space. This structural inductive bias encourages the TP-TRANSFORMER to encode the syntactic information in the discrete roles while isolating the semantics in the continuous fillers. To test the ability of our TP-TRANSFORMER with discrete roles against the standard Transformer and the TP-TRANSFORMER with continuous roles, we build several models from scratch on a number of summarization datasets spanning different degrees of abstractiveness, output summary lengths, and domains. Our TP-TRANSFORMER significantly outperforms the standard Transformer and the TP-TRANSFORMER with continuous roles on the XSum (Narayan et al., 2018), Wikihow (Koupaee and Wang, 2018), and Arxiv (Co-han et al., 2018) datasets and achieves competitive performance on the CNN/Daily Mail (Hermann et al., 2015; Nallapati et al., 2016) dataset, measured by automatic metrics including ROUGE (Lin, 2004) and METEOR (Denkowski and Lavie, 2014). Our human evaluations on XSum and Wikihow datasets also correlate with the automatic metrics, demonstrating that summaries generated by our TPTRANSFORMER are indeed better than the Transformer's generations. Furthermore, to investigate the structural representation that naturally emerges during training and the advantage of having compositional TPR hidden states, we design a suite of decoder probing tasks to explore the information encoded in the role, filler, and TPR space. We adopt the encoder probing task design presented in Tenney et al. (2019b) and create four decoder probing tasks: Part-of-speech tagging (POS), Dependency Labeling (DEP), Semantic Role Labeling (SRL), and Named Entity Labeling (NEL). Our findings collectively show that the decoder's role vectors encode a wealth of syntactic structures, aiding the decoder in deducing the syntactic features (e.g., being a proper noun, being the object of the root predicate) of the next token to be generated. The decoder's filler vectors on the other hand encode more semantic information (e.g., being a person's name). Furthermore, we observe that having the compositional TPR results in a more interpretable final representation than the original Transformer has at every layer, regarding the syntactic features of the next word to be generated. Our results support our hypothesis that by disentangling semantics and syntax, such structured intermediate representations enable the model to better control both the content to be conveyed and the syntactic structure needed to express it, ultimately improving the factuality and grammaticality of the generated summaries. Our overall contributions are as follows: (1) we present a novel adaptation of the original Transformer architecture that incorporates a dictionary of role embeddings at every layer and generates Tensor Product Representation by binding the role vectors with attention outputs (filler vec-tors); (2) show that our TP-TRANSFORMER outperforms the Transformer as well as the original TP-TRANSFORMER (Schlag et al., 2019) on several abstractive summarization datasets; and (3) demonstrate the emergent structures in representations by revealing the disentangled syntactic and semantic information encoded in the role and filler spaces. 2 The TP-TRANSFORMER We build our TP-TRANSFORMER based on the Transformer architecture used in Raffel et al. (2020). A TP-TRANSFORMER encoder applied to a sequence of tokens i = 1 , ..., I can be seen as a 2-dimensional lattice of cells ( i, l ) where i is the Lin ear V K Q Scaled Do t-Pro du ct Atten tio n Lin ear Lin ear Lin ear Ro le Em beddin gs Ro le-Atten tio n M ulti -H ead A tt e ntion Lin ear TPR Filler s (F) Ro les (R) Co n cat Co n cat Figure 2: The Filler and Role Binding operation of the TPTRANSFORMER Model architecture. position of the input token and l = 1 , ..., L are the layer indices. All cells in the encoder have the same architecture and the cells at the same layer share the same weights. We introduce the basic components of a TP-TRANSFORMER cell in Sec. 2.2 and its encoder and decoder cells in Sec. 2.3. 2.1 Tensor-Product Representation Basics Tensor-Product Representations (TPR; (Smolen-sky, 1990)) are explicitly-compositional vector embeddings of symbolic structures, where each constituent of the structure is represented as the product of a role vector, which encodes its structural information, and a filler vector, which contains the content. The TPR of a whole structure is the sum of the representation of its constituents. To represent any 3-digit number using TPRs, we need three role vectors: { r ( p 1) : Ones place, r ( p 2) : Tens place, r ( p 3) : Hundreds place } and ten filler vectors f for ten digits. For example, the TPR of the number 985 is r ( p 1)  f (5)+ r ( p 2)  f (8)+ r ( p 3)  f (9) , where  is the tensor product. When representing a number, the role vectors operate similarly as the positional embeddings in a Transformer (Vaswani et al., 2017). However, when representing natural languages, the role vectors need to encode a variety of structural information (e.g., predicate-argument, tense, etc) and thus it is infeasible to hand-design an entire suite of role vectors as we did for numbers. To overcome this challenge, for every token, we dynamically compute its role vector from a dictionary of a finite number of role embeddings learned with the entire model and treat the self-attention outputs as the fillers. We introduce the full computation procedure in Sec. 2.2.2. 2.2 The TP-TRANSFORMER Cell Similar to the basic Transformer cell, at every layer, a TP-TRANSFORMER Encoder cell starts with a layer normalization and the multi-head self-attention followed by a residual layer. Then, the cell treats the output vectors as fillers and binds them to role vectors to construct a Tensor Product Representation, which is then passed through the feed-forward network to yield the final states. 2.2.1 Multi-Head Attention The TP-TRANSFORMER cell adopts multi-head attention (Vaswani et al., 2017) to enable information passing between tokens. At any layer, denote the input vectors as X  R k x  d m and the attention target vectors as Y  R k y  d m , where k x , k y are the length of the sequences and d m is the dimension of the input vectors. In the case of self attention, we have Y = X ; while for the encoder-decoder cross attention, Y is the encoder's output vectors. We first apply layer normalization (Ba et al., 2016) to get  X and then linearly project it to the query, key, and value vectors for each attention head h = 1 , ..., H . Q h =  X W hq + b hq K h = Y W hk + b hk V h = Y W hv + b hv (1) where W q , W k , W v  R d m  d k . The attention output matrix  V for each head h is computed as:  V = softmax( QKT  d k ) V (2) where d k is the dimension of the key vectors K . The multi-head attention output O is the concatenation of the attention outputs from all heads followed by another linear projection W o  R d m  d m . We end the Multi-head Attention with a residual connection with the layer input vectors  X : MHAttn( X, Y ) =  X + [  V 1 , ...,  VH ] W o (3) where  V h is the attention output for the h -th head. 2.2.2 Computing TPRs Role Embeddings. Following Palangi et al. (2018), but departing from Schlag et al. (2019), every layer of our TP-TRANSFORMER is equipped with a dictionary r  RN r  d r of N r distinct role embeddings with a dimension of d r . Each role embedding r n , n =1, . . . , N r , is randomly initialized in the entire network. The role embeddings are normalized before computing role vectors:  r n = r n (cid:107) r n (cid:107) 2 for n = 1 , ..., N r (4) At each layer, the model computes a weighted combination of these role embeddings  r to form a unique role vector for every token. Multi-Head TPR Binding. Our filler vectors correspond to the multi-head attention output F = MHAttn( X ) (Eqn. 3). The filler F of each token has a corresponding role vector R . We first compute the R h  R d r at every head h = 1 , ..., H as a weighted average of the normalized role embeddings  r . We then concatenate the R h  R k x  d r of H heads to get the multi-head role vectors R  R k x  ( d r  H ) for all k x tokens. We define this process formally as: R h = softmax( F W hr ) r R = [ R 1 , ..., RH ] (5) where W r  R d m  N r is the linear projection that computes the attention scores over the role embeddings for every token. 2 We use a Hadamard product 3 to approximate the full Tensor product in binding the role vectors R with filler vectors F , as it was shown in Schlag et al. (2019) that using the Hadamard products allows learning an optimial lower-rank approximation of the full TPRs. The binding operation is followed by an addition with the unbound fillers ( F ) to return the residual TPR vectors. TPR( F ) = R (cid:12) F + F (6) 2.2.3 Residual Feed-forward Layer The feed-forward layer of a cell consists of a linear projection followed by a ReLU activation and a second linear projection. The feed-forward output is then added to the input vectors: FF( X ) = X +ReLU( X W g + b g ) W f + b f (7) Here, W g  R d m  d f , b g  R d f , W f  R d f  d m , b f  R d m , and x is the function argument. 2 We set d r  H = d m so that the multi-head role vectors R have the same dimension as F . 3 The Hadamard (or elementwise) product is the diagonal of the full tensor product. 2.3 TP-TRANSFORMER Encoder & Decoder Given the components of our basic TPTRANSFORMER cell in the previous section, we now describe how we construct the TPTRANSFORMER encoder and decoder. First, the self-attention and the encoder-decoder cross-attention for every token can be computed as: Self( X ) = TPR(MHAttn( X, X )) Cross( Y, H ) = TPR(MHAttn( Y, H )) (8) where H is the output of the encoder's final layer. Y represent the previous layer's output vectors of either the partially (so-far) decoded sequence at test time or the masked reference summary at training time. The encoder and decoder's operations at every layer can be summarized as: Encode( X ) = FF(Self( X )) Decode( H, Y ) = FF(Cross(Self( Y ) , H )) (9) After L layers of encoding and decoding, the final distribution of the i -th output token is given by:  z i = softmax( ET y i,L ) (10) where YL = Decode( H, YL  1 ) are the decoder's output states at the last layer and E is the tied input/output word embeddings. 3 Summarization Experiments 3.1 Abstractive Summarization Datasets We train our models on four English abstractive summarization datasets varying the level of abstractiveness (explained below) and the length of summaries, as well as input domain. XSum (Narayan et al., 2018) consists of 227k BBC articles from 2010 to 2017 concerning various subjects along with professionally written single-sentence summaries. Its summaries cover a wide variety of syntactic structures (relative clause, etc) and relations (causal, temporal, etc). Wikihow (Koupaee and Wang, 2018) is a dataset consisting of instructions from the WikiHow.com website. Each of 200k examples has multiple instruction-step paragraphs, each paired with a summarizing sentence. The task is to generate the concatenated summaries of all paragraphs. Datasets Summary XSum Luxury fashion designer Burberry has returned to profit after opening new stores and spending more on online marketing. Wikihow Build a trustworthy bond with your piggy. Research different training methods. Choose the training method that works best for you and your guinea pig. Gather the materials that you will need for training. Arxiv (Abbreviated) We study the phase behavior of a nematic liquid crystal confined between a flat substrate with strong anchoring and a patterned substrate whose structure and local anchoring strength we vary. [...] In addition the effective energy method allows one to determine the energy barriers between two states in a bistable nematic device . CNN/DM Mentally ill inmates in Miami are housed on the "forgotten floor". Judge Steven Leifman says most are there as a result of "avoidable felonies". While CNN tours facility, patient shouts: "I am the son of the president". Arxiv (Cohan et al., 2018) is a long document summarization dataset of scientific publications from arXiv.org (113k). The task is to generate the abstract from the paper body. CNN/Daily Mail (Hermann et al., 2015; Nallapati et al., 2016) dataset contains 93k articles from CNN and 220k articles from the Daily Mail. Every article is accompanied by a few human-written bullet points about its content. We use the non-anonymized version used in See et al. (2017). Dataset Abstractiveness. We show a summary from each of these four datasets in Table 1. According to the comparison made by Zhang et al. (2020) using the coverage and density measures (Grusky et al., 2018), the XSum and Wikihow datasets are more abstractive than the others since their summaries rarely contain large chunks of words overlapping with the source documents. CNN/Daily Mail is the least abstractive of the four. Furthermore, in most cases, a sentence in a CNN/Daily Mail summary only refers to a single sentence from the source document as suggested in Lebanoff et al. (2019), while a sentence in an XSum or Wikihow summary usually aggregates information from multiple source sentences. The Transformer and the two TP-TRANSFORMERS all have 6 layers, 8 heads per layer, dimension per head d k =64, model dimension d m =512, and feed-forward dimension d f =2048 for the encoder and decoder. Our TP-TRANSFORMER with discrete roles has N r =50 role embeddings of dimension d r =64 at every layer. For each dataset above, we train the all three models from scratch using an Adafactor Optimizer (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.1. We evaluate the models using automatic metrics including ROUGE F1 score and METEOR. We report automatic metric scores from our evaluated models in Table 2. We refer to the TPTRANSFORMER , with freely-generated continuous role vectors (no role dictionary) (Schlag et al., Datasets Models Grammar Coherency Faithfulness Saliency Repetition Overall XSum Transformer wins 39 48 43 50 38 48 TP-TRANSFORMER wins 47 48 46 47 42 52 Tie / No agreement 34 24 31 23 40 20 Wikihow Transformer wins 45 45 43 54 48 43 TP-TRANSFORMER wins 48 45 46 47 48 59 Tie / No agreement 27 30 31 19 24 18 Table 3: Human Evaluation results on 120 random samples from the XSum (Narayan et al., 2018) and Wikihow (Koupaee and Wang, 2018) test sets. 2019) as TPT-c, and our own TP-TRANSFORMER with a discrete set of role embeddings as TPT-d. On the XSum, Arxiv, and Wikihow datasets, our TP-TRANSFORMER (TPT-d) outperforms the original Transformer on all metrics. On the CNN/Daily Mail dataset, both models obtain similar performance across all metrics. On every dataset, the TPT-c model which excels on the mathematics dataset, is the worst among the three models being compared. This suggests that continuous role vectors are not suited to the summarization tasks. As we explain in Sec. 3.1, CNN/Daily Mail is the least abstractive one among the four datasets. In contrast, summaries from the XSum and Wikihow datasets contain very few n-grams (n > 2) that can be copied from the source documents and thus push the model's ability to compose a coherent summary restating the salient aspects from the source. Furthermore, as illustrated in Table 1, the XSum summary contains a long sentence that combines multiple pieces of information scattered through the long source document. These facts are usually connected by syntactic, temporal 4 , or causal 5 relations and thus the model must be able to connect and reason across these salient facts and then convert them into a coherent sentence that faithfully reflects the original facts and their relations. We argue that the compositional TPR can better enable these abilities required for XSum, where we indeed find that our TP-TRANSFORMER achieves the largest advantage over the Transformer among its improvements on all datasets. We conduct human evaluation to compare the summaries generated by the Transformer and our TPTRANSFORMER . We randomly sample 120 examples from the test sets of XSum and Wikihow datasets with the beam-searched model summaries. 4 returned to profit after opening new stores" 5 Opening new stores and spending more on online marketing" caused "more profit". We refer to appendix for the complete setup. As shown in Table 3, on the XSum dataset, summaries generated by the TP-TRANSFORMER are significantly better in grammar. This corroborates our claim that having the TPR can improve the model's ability to follow the correct syntax in composing the summary. On the Wikihow dataset, the Transformer receives more votes in regarding the saliency. However, our TP-TRANSFORMER maintains an advantage in grammar and achieves significantly better overall preferences. Unfaithful XSum Examples It is well-known that the XSum dataset contains a portion of unfaithful reference summaries that mention facts not included in the source article (Durmus et al., 2020; Maynez et al., 2020). Therefore, we are interested to find out whether our TP-TRANSFORMER is better than the baseline only at expressing the faithful content or it can also generate some external, un-faithful" facts that the baseline can't cover. To answer this question, we randomly sample 100 examples from the XSum dev set and manually examine the source document, reference summary, and the two generated summaries. Among these 100 examples, we identify 71 examples whose reference summary includes unfaithful" facts that are not mentioned in the source. In 21 out of 71 examples, the Transformer baseline manages to generate some unfaithful" facts that match those in the reference while our TP-TRANSFORMER achieves this in 17 examples. Such unfaithful" facts that were recovered by the models include the full name of a person when only the last name is mentioned in the source, the political party or the job title of a person, each of which can be attributed to at least one example seen by models during the training. Therefore, we believe that both models learn to draw external information from its memory of the seen examples, while our TP-TRANSFORMER doesn't do better than the baseline Transformer at referring to external facts to obtain higher ROUGE scores. Probing is a method to test whether some particular information is present in the model's encodings. To achieve this, an auxiliary classifier is trained to predict specified linguistic features from the model's internal representations. We probe different components (roles, filler, TPRs) in our TPTRANSFORMER s as well as the attention+residual outputs (equivalent to the filler) of the Transformer to assess the naturally emergent structures encoded in the role vectors and the effectiveness of the TPR in the decoding process. By conducting the probing experiments, we aim to (1) provide some insights and evidence of the different information encoded by the role and filler vectors; and (2) explain the ROUGE advantage of our TP-TRANSFORMER by showing that its output representation can better encode the linguistic structural information concerning multiple probing tasks. When studying an encoder, previous works probe its i -th intermediate representation at a certain layer for information about the i -th input token For a decoder, however, we probe its i -th representation for clues about the i -th token it generates given the i  1 previously generated tokens as the input. Intuitively, we are probing for the decoder's internal decision about the syntactic roles and semantic content of this token before it was ultimately selected. Based on encoder probing tasks used by Tenney et al. (2019b), we select and adapt four tasks to probe our decoders. Part-of-speech tagging (POS) is the syntactic task of assigning tags such as noun (singular/mass noun: NN, proper noun: NNP, etc), verb (past tense: VBD, past participle: VBN, etc), adjective (comparative: JJR, etc), etc. to each token i . We let s 1 = [ i, i + 1) be a single token, and seek to predict its POS tag. Dependency labeling (DEP) seeks to predict the functional relationships of one token relative to another: e.g. is it a modifier-head relationship, a subject-verb relationship, etc. We take s 1 = [ i, i + 1) to be a single token and s 2 = [ j, j + 1) to be its syntactic head, and seek to predict the dependency relation between tokens i and j . imposing predicate-argument structure onto a sentence. We let s 1 = [ i 1 , j 1 ) represent a known Tasks Layer Transformer TPT-d (Ours) POS 1 -/ 58.4 /58.4 36.1/57.1/58.2 2 -/ 65.4 / 65.4 43.6/63.5/64.4 3 -/ 68.6 /68.3 50.4/67.4/68.5 4 -/70.7/70.7 50.4/70.8/ 72.1 5 -/72.5/72.5 53.4/ 73.3 / 73.9 6 -/73.3/73.3 56.0/ 73.9 / 74.5 DEP 1 -/78.1/78.1 53.1/ 78.8 / 78.9 2 -/85.0/85.0 59.9/84.8/84.7 3 -/87.1/87.1 66.7/87.4/87.3 4 -/87.4/87.4 62.9/ 88.3 / 88.2 5 -/85.0/85.0 64.8/ 88.3 / 87.6 6 -/86.1/86.1 60.8/ 86.8 / 86.6 SRL 1 -/78.2/78.2 73.1/78.5/78.4 2 -/79.0/79.0 73.8/ 79.8 /79.3 3 -/79.6/79.6 73.8/79.9/80.0 4 -/78.7/78.7 73.1/ 80.1 / 80.2 5 -/77.7/77.7 72.9/ 79.9 / 79.8 6 -/78.1/78.1 71.8/ 79.2 /78.2 NEL 1 -/59.7/59.7 33.3/ 61.4 / 60.8 2 -/67.6/67.6 37.6/ 68.1 / 68.2 3 -/69.6/69.6 41.5/ 70.9 / 71.0 4 -/71.8/71.8 43.6/ 74.3 / 73.2 5 -/72.3/72.3 44.7/ 76.3 / 75.7 6 -/73.3/73.3 42.2/ 76.1 / 73.8 Table 4: Results (F1 scores) of probing different intermediate representations in decoders trained on XSum dataset. predicate (e.g., push") and s 2 = [ i 2 , j 2 ) represent a known argument (Peter") of that predicate, and seek to predict the role that the argument s 2 fillse.g. ARG0 (agent, the pusher) vs. ARG1 (patient, the pushee). Named entity labeling (NEL) is the task of predicting the category of an entity. The categories include PERSON , LOCATION , ORGANIZATION , etc. We let s 1 = [ i, j ) represent a known entity span and seek to predict its type. As there is no existing dataset for probing decoders, we create our own training and evaluation data by running off-the-shelf models on the summarization datasets. Specifically, to probe a decoder trained on the XSum dataset on the POS task, we run an POS tagger on the reference summaries from the XSum training set and the model-generated summaries for the XSum dev set to create the ground-truth labels for the training set and model-specific dev set. We restore the model trained on a summarization dataset and freeze its parameters. Following Tenney et al. (2019b), we train a span convolution layer followed by a 2-layer MLP on top of the target representation that project it onto the output label space. Table 4 presents the results of probing the decoder of a TP-TRANSFORMER trained on the XSum (Narayan et al., 2018) dataset. Note that the Transformer doesn't have role vectors. It directly outputs the vector after the multi-head attention and the residual layer. Therefore, its fillers and final representations are equivalent. The decoder role vectors can encode grammatical information while the filler vectors represent the semantics. We first focus on the results of POS tagging probing task. Overall, we see a trend of increasing scores as the representations get closer to the final step of computing the distribution over the vocabulary. This implies that, as the computation progresses through the layers, the generated representations are gradually deciding the POS tag of the next word to generate. Next, we observe that the role vectors (the 1st number in the TPT-d column) of TP-TRANSFORMER encode a considerable amount of information about the POS tag of the next word generated. Additionally, because the job of deducing the POS tag of the next word is partially shared by the role vectors, the filler vectors' performance degrades compared to the Transformer. This pattern demonstrates that the TP-TRANSFORMER 's decoder is representing the next word to be generated as a composite of structural information encoded in the role vectors and semantic contents encoded in the filler vectors. Comparing the fillers (the 2nd number in TPT-d column) with the TPR (the 3rd number in the TPT-d column) of TP-TRANSFORMER , we see that the TPRs, which bind the roles and fillers, outperform the roles and fillers alone at every layer. This indicates that the TPR effectively aggregates the linguistic knowledge encoded in the roles and fillers into a shared space, where the POS tag of the next word can be decoded more easily than in the role space or filler space alone. Last, the final representations of TP-TRANSFORMER achieve higher F1 scores than their counterparts in the Transformer in the last three layers. This demonstrates the benefits of having the TPR in interpreting the POS tag of the word to be generated. When we consider the Dependency labeling (DEP) and Semantic role labeling (SRL) tasks, we observe that our TP-TRANSFORMER 's final representations consistently beat the Transformer across all layers, with only one exception in the DEP task at the layer 2. We also observe that the TP-TRANSFORMER 's advantage becomes larger in the last three layers except for the final layer in SRL task. However, unlike in the POS task, the TPR only achieve similar F1 scores to the fillers. Finally, in the Named entity labeling (NEL) task which is considered to require more semantic information rather than syntax, the role vectors' performance is poorer than their performance in the three syntactic tasks. For example, the TPTRANSFORMER 's final representations at layer 6 obtain similar F1 scores in the POS and NEL tasks (74.5 VS 73.8), but its role vectors only achieve a 42.2 F1 score in the NEL tasks compared to the 56.0 in the POS. However, even though the role vectors encode little information about the named entity type of the next token to be generated, the TPR still strongly outperforms the Transformer's filler-only representation at every layer. We argue that although the syntactic information encoded in the role vectors is not enough to predict the correct named entity, it is still a beneficial complement to the knowledge encoded in the distributed filler vectors in certain situations. For example, whether the subject Chanel" refers to a PERSON or an ORGANIZATION could depend on its syntactic role and its relation to other words in the sentence (e.g., whether it is the subject or object of wears) . interpretability of the representations. Overall, by probing the different intermediate representations of the TP-TRANSFORMER and the Transformer, we show that having the compositional TPR results in more interpretable final representations at every layer regarding the syntactic features of the next word to be generated. Considering automatic evaluations generated summaries in Sec. 3.3, we argue that this compositionality in learned representation and its syntactic interpretability enable the decoder to take better control of the syntactic structure of the generation when assembling multiple distant facts, and thus lead to summaries of better quality. During the training of our TP-TRANSFORMER models on the summarization datasets, we observe that most learned role attention distributions are approximately one-hot, as more than 90% of the role attention distributions (as computed in Eqn. 5) have a maximum score larger than 0.98. Because each role vector is the concatenation of H vectors, each selected from N r role embeddings, the completely one-hot role attentions will yield ( N r ) H possible role vectors. Therefore, the learned, approximately one-hot role vectors span ( N r ) H discrete subspaces, each of which only covers the close proximity of a concatenation of H role embeddings. This finding indicates that as we represent the role vectors as multi-head attention over a learnable dictionary of role embeddings, the structural inductive bias: (1) pushes the role vector space to be even more discrete, and (2) induces the syntactic structures encoded in these discrete role vectors. We also believe there is a connection between the above two effects, as the structural, syntactic information favors a lower-dimensional or even discrete space while the distributed, semantic information favors a higher-dimensional space. Explicit TPR Structures in Neural Networks While earlier TPR work based on (Smolensky, 1990) focused on computability rather than learn-ability questions, recently TPRs have been incorporated into several recurrent deep learning models in order to solve various NLP tasks including Part-of-Speech tagging, constituency parsing, image captioning (Huang et al., 2018, 2019), question answering (Palangi et al., 2018; Schlag and Schmid-huber, 2018), and natural-to-formal language generation (program synthesis) (Chen et al., 2020). Most recently, TPRs have been introduced into Transformer architectures, starting with Schlag et al. (2019) which introduced the TP-TRANSFORMER to improve the performance and interpretability of mathematical problem solving models. This model generated continuous role vectors by directly projecting from layer inputs, whereas our model indexes from a dictionary of role embeddings to form the role vectors which are shown to reside in a highly discrete space. Structured Representations for Abstractive Summarization Compared to the extractive methods, abstractive summarization models usually fail to show extractive properties, and have tendency to copy text from the source (See et al., 2017; Paulus et al., 2018; Pasunuru and Bansal, 2018; Celikyilmaz et al., 2018). More recent approaches that use standard transformers deal with this issue by introducing hierarchical structures to encode local and global information separately focusing on only the semantic content (Liu and Lapata, 2018, 2019). To preserve salient source relations and generate abstractive summaries of the source document, previous work infused models with semantic parsers: while Song et al. (2018) introduces a new structure-infused copy mechanism that combines the source syntactic structure with the copy mechanism, Liao et al. (2018) uses abstract meaning representations (AMR). While these approaches require that the document sentence semantic parsers are provided beforehand, our models can implicitly learn to approximate the syntactic structure and semantic content in their representations. In this work, we enrich the Transformer model with the structured Tensor Product Representation for abstractive summarization tasks. We represent every token as a pair of role and filler vectors. We show that our TP-TRANSFORMER with discrete roles outperforms Transformer and TPTRANSFORMER with continuous roles on several abstractive summarization datasets, in both metrics scores and human evaluation. We further demonstrate the syntactic structures encoded in the role vectors and show the improved syntactic interpretability in our model's hidden states. In this work we propose a new encoder-decoder modeling architecture and build several models to benchmark our new architecture with baseline architectures on several open source summarization datasets. Intended use. Our architecture is designed to build models of abstractive summarization. Potentially our architecture could be used to train models for summarizing any type of company internal datasets (e.g., internal documents, reports, meetings, legal forms, etc.) to further improve the productivity and efficiency of the users in their daily activities without needing to read long documents. Failure mode. Even though our models yield factually consistent summaries, as judged by human evaluation, they can still generate factually inconsistent summaries or sometimes hallucinate information that the source document does not include. This might be due to the bias or noise in the training data. Model builders wanting to use our architecture to build models on their company internal datasets should build models with consideration of intellectual properties and privacy rights. Misuse Potential. We note the models to be built with our architecture should be used with careful consideration. The generated summaries produced by our models are not controlled and use generative approaches, therefore, they could generate unreliable text. Researchers working on abstractive summarization should focus on generating factually correct, ethical and reliable text. If our models are trained on news datasets, a careful consideration should be made on factuality of the generated text and measures have been taken to prevent model hallucinations. We thank the reviewers for their helpful comments. This work was partially supported by NSF-CAREER Award 1846185 and a Microsoft Investigator Fellowship.
abstain abstain method abstain method result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain method abstain result abstain abstain abstain abstain method method method abstain result abstain other abstain abstain abstain abstain abstain abstain abstain result method method abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method method method method method abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain method result abstain abstain abstain abstain other other abstain other other other method abstain abstain result objective abstain abstain method abstain abstain method abstain method abstain method method abstain method other other