Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections. We propose READTWICE 1 , a simple and effective technique that combines several strengths of prior approaches to model long-range dependencies with Transformers. The main idea is to read text in small segments, in parallel, summarizing each segment into a memory table to be used in a second read of the text. We show that the method outperforms models of comparable size on several question answering (QA) datasets and sets a new state of the art on the challenging NarrativeQA task, with questions about entire books. Transformer-based models such as BERT are very effective in capturing long-range dependencies in text passages through the attention mechanism (Vaswani et al., 2017; Devlin et al., 2019). However, the amount of compute in attention depends quadratically on the number of tokens in an input text passage. As such, the standard BERT implementation limits input size to a fixed number (often 512) of tokens. In reality, dependencies over significantly longer ranges are common and modeling them is crucial. For instance, in a sentence like Inside the Sammath Naur, the Ring-bearer struggled to throw the Ring into the volcano , the narrative interweaves several prior storylines from a book. Comprehending this sentence therefore requires looking up previous  Work is done while at Google  On leave from University of Southern California (feisha@usc.edu) 1 Source code and pre-trained checkpoints for READTWICE can be found at https://goo.gle/ research-readtwice . Several methods have been proposed to address this challenge; see (Tay et al., 2020) for a survey and 3 for a detailed discussion. One popular strategy is to reduce the number of tokens attended to. Longer inputs can in fact be processed in this way  but only up to a limit of around 5,000 tokens, as used in (Ainslie et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020)  far below the context sizes required to model long documents such as books. Another strategy such as HIBERT (Zhang et al., 2019) splits inputs into smaller segments which are processed individually, then assembled into a hierarchical representation. As a downside, inter-segment context is unavailable during encoding. We propose READTWICE , a simple approach that combines the strengths of both strategies. As its name suggests, the main idea is to process the input twice: a long text input (such as a document, or even a book) is treated as a collection of shorter text segments which are read independently and in parallel. Then, the encoder reads each segement again, now augmented with compressed information from other segments. The crucial component in READTWICE , as illustrated in Figure 1, is a memory module that holds compressed information from all segments. That compressed information is used only once : in the second pass. Thus, READTWICE is much more computationally efficient than models like ETC that rely on memory for all segments, in every layer. While READTWICE requires two passes, it differs from hierarchical models such as HIBERT that do not condition segment encoding on other segments. 3 contrasts these approaches in more detail. We validate the efficacy of READTWICE on extractive question answering (QA) tasks, showing strong performance on HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017) and Narra-Figure 1: READTWICE model architecture. The input is processed twice, with a memory table for inter-segment information sharing. tiveQA (Kocisk et al., 2018). In particular, READTWICE significantly improves the state-of-the-art on QA based on entire books in NarrativeQA, with absolutes gains of 4.5 ROUGE-L points and 3 BLEU-1 points (relative improvements of 23% and 17%, respectively). The model reads a large text document split into N segments x 1 , . . . , x N ; each x i is limited to 512 tokens, as in a typical BERT model. The model architecture is depicted in Figure 1. In the first read, each segment is encoded independently with standard BERT. Then, memories are extracted from each segmenta process we describe in detail laterand gathered into a global memory pool. For the second read, a MemoryAttention layer (with a residual connection and a LayerNorm on top) is first used to merge the information from the former intra-segmental contextual token embeddings and the global memory. The merged result is then read by another small BERT model with only two Transformer layers to produce the final output. The rationale is that the first read already generates rich contextualized embeddings, and the second read only needs to incorporate information from the memory. More formally: H 0 i = TokenEmbed ( x i ) , H 1 i = BERT 1 ( x i ) ,  i M i = ExtractMemories ( H 1 i ) ,  i M = Gather ([ M 1 , . . . , MN ]) H 2 i = MemoryAttention ( H 1 i , M ) ,  i H 3 i = LayerNorm ( H 1 i + H 2 i ) ,  i H 4 i = BERT 2 ( H 3 i ) ,  i Next, we describe the newly introduced layers. ExtractMemories and Gather Our aim is to compress the information in each segment and disseminate it to other segments to be used in the second read. We consider three types of memories:  READTWICE ( CLS ). One obvious choice is to use the CLS token representation associated with segment x i as a summary of the segment. READTWICE ( STS ). To obtain more fine-grained memories, we extract a memory vector for each consecutive span of 32 tokens. Contextual embeddings of each span's first and the last tokens are concatenated and linearly projected to a single point in the token vector space as the span representation. The projection matrix is learned end to end. READTWICE ( E ). In another variant of span-based memory, we memorize representations of entity mention spans. To obtain these spans, we first annotate each segment with an external Named Entity Recognition system. Then, each entity mention span is encoded in the same way as in READTWICE ( STS ). This design is motivated by the intuition that long-range dependencies primarily occur between entities. Empirically, we find that READTWICE ( E ) leads to best performance (see the ablation in Section 4.4) and it is the memory type used in our headline results. We collect all memories from all segments into a flat memory table. The table size is given by the number of segments ( CLS ), the number of 32-token spans ( STS ), or the number of entity mentions ( E ). MemoryAttention In this layer, we let contextual token embeddings from individual segments interact with other segments' memories via dot-product attention over the memory table. where M 0 is a learnable no-op memory not associated with any specific text. r i,m s is a learned position score which captures the relative distance between segment i and the memory M m , akin to Shaw et al. (2018): r i,m s =  ( dist ( i, m s )) (2) where  is a set of weights indexed by the distance dist ( i, m s ) =    B i  m s <  B B i  m s > B i  m s otherwise (3) where the cutoff threshold B clips the effect of distance to [  B, B ] . We set B to 10 in this work. Finally, the MemoryAttention layer output for a given token is given by h 2 ij = (cid:88) m =1  m M m (4) 2.2 Pre-training We pretrain READTWICE similarly to (Devlin et al., 2019), using the Wikipedia and BooksCor-pus datasets. When entity mentions are used in the memory table, the texts are processed with the Entity Linking (EL) and Named Entity Recognition (NER) tools from the Google Cloud NLP API 2 . Moreover, we use existing hyperlinks in Wikipedia as additional entity annotations. The first and the second BERT readers are trained end-to-end. Our pre-training objective is the standard Masked Language Model (MLM) task, with the MLM prediction loss computed based on the output of the second reader. In order to encourage the model to rely on the memory, we increase the difficulty of the MLM task. Following the entity masking procedure in (Guu et al., 2020; Sun et al., 2019), we mask entity mention tokens more aggressively at a 25% rate and jointly mask all tokens within a mention. By contrast, for non-entity tokens, we mask contiguous sequences of random length at a 15% rate. One way to extend the limit on input size is by reducing the number of tokens attended to. ETC (Ainslie et al., 2020) and LONGFORMER (Belt-agy et al., 2020) allow standard attention only between tokens within a fixed distance. To allow information flow over longer distances, they use auxiliary global "memory" tokens which attend to all regular tokens and vice versa. BIGBIRD (Zaheer et al., 2020) additionally has each token attend to a random subset of other tokens. While reducing asymptotic complexity from quadratic to linear (in input size), these global tokens are added at each attention layer, incurring a high computational cost. Another approach is to split the input into multiple segments and then aggregate information across segments. This is achieved through hierarchical modeling (Chang et al., 2019; Zhang et al., 2019). While reducing the attention size to the number of segments, each individual segment has no information about its siblings during token-level encoding. Alternatively, recurrent models (Dai et al., 2019; Rae et al., 2019) read a large input from left to right, dynamically compressing faraway contexts, thus allowing unidirectional information aggregation (left to right). One disadvantage is that the input needs to be processed sequentially, which becomes time-consuming for producing contextualized representations of a large input. Our method brings these lines of work together. Processing segments independently and in parallel, then memorizing their compressed representations and sharing memory across segments enables contextual embeddings to be updated based on faraway information. Enabling memory sharing only once during the second readallows it be done cheaply. Note that the memory module here is internally generated from the input, as opposed to external memory models which are orthogonal to our approach (Peters et al., 2019; Fvry et al., 2020). All READTWICE models are initialized with the public ROBERTA (base) checkpoint 3 adapted to Tensorflow by Rothe et al. (2020). Further, models are pre-trained for 1M steps on 64 TPU cores using the LAMB optimizer (You et al., 2020). Each batch contains 512 segments, with at most 128 segments per document. The segments are consecutive spans of 512 tokens. Therefore, the model can process documents up to 65k (  128  512 ) tokens. Each batch contains the maximum number of documents such that the total number of segments is at most 512. Approximately half of Wikipedia articles fit in one segment (thus not needing memory), with a fat tail of longer documents. In terms of compute and memory overhead, READTWICE is about 30% slower than the ROBERTA -base model and uses 15M (or 12%) more parameters: 14M owing to the second read BERT 2 and 1M due to ExtractMemories and MemoryAttention layers. We evaluate READTWICE on the downstream extractive question-answering task using several datasets: HotpotQA (HQA) (Yang et al., 2018), TriviaQA (TQA) (Joshi et al., 2017) and NarrativeQA (NQA) (Kocisk et al., 2018). In HQA, questions are based on relatively short text passages (2 evidence paragraphs), with eight additional distractor passages. In TQA, evidence text is medium-sized. NQA asks questions about entire books, requiring a successful QA system to model very long-range dependencies. The NQA dataset has an average of 62,000 words per document with a maximum of 400,000. Only 40% of NQA's answers are span-based  we use a ROUGE-L oracle as training labels for the other questions. READTWICE is fine-tuned on each task. QA-specific heads are used to generate span-based predictions, consisting of fully-connected layers that take contextual embeddings from the second reader as inputs. These layers output a score for whether the corresponding tokens are the beginning or ending of an answer span. For a similar setup, see multi-segment based QA tasks (Clark and Gardner, 2018; Cheng et al., 2020). During fine-tuning, batches contain 128 segments for all tasks (also with up to 128 segments per document). Every segment contains 512 tokens, but as neighboring segments have 128 token overlaps, the model can process documents of up to 49K tokens (  128  (512  128) ). For TQA and HQA, documents have approximately 10 segments. For NQA, we split the documents into sub-documents with 49k tokens and apply memory only within these sub-documents. Moreover, we use early stopping based on the performance on the development set. Results for HQA and TQA are reported in Table 1. We compare to prior art (using reported results where available or from our own implementations otherwise, denoted as us): Longformer (LF) (Beltagy et al., 2020), ETC (Ainslie et al., 2020), BigBird (Zaheer et al., 2020), and ROBERTA (Liu et al., 2019). By default, we compare against the base configuration of those models where the number of parameters is comparable to BERT-Base, as is the case for READTWICE . Table 1 shows that for small to medium sized text passages, the proposed READTWICE outperforms all models of comparable size. Table 2 contrasts READTWICE to other methods on extremely large contexts: BiDAF (Kocisk et al., 2018), R 3 (Wang et al., 2018), BM25 + BERT Reader / Ranker (Mou et al., 2020) and our own implementation of ROBERTA and ETC 5 . READTWICE significantly outperforms all previous work and establishes new state-of-the-art results, demonstrating the effectiveness of performing a second read conditioned on global memory for processing extremely long texts. 4.4 Ablation Analysis & Discussion To isolate individual components' contributions, Table 3 contrasts several variants of READTWICE . 4 See https://competitions.codalab.org/ competitions/17208#results , tab Wikipedia. 5 For ETC we use the public (base configuration) checkpoint https://storage.googleapis.com/ gresearch/etcmodel/checkpoints/etc_base_2x_pretrain.zip Model ROUGE-L BLEU-1 BLEU-4 METEOR BiDAF (Kocisk et al., 2018) 6.3 / 6.2 5.8 / 5.7 0.2 / 0.3 3.8 / 3.7 R 3 (Wang et al., 2018) 11.4 / 11.9 16.4 / 15.7 0.5 / 0.5 3.5 / 3.5 BM25+BERT (Mou et al., 2020) 14.8 / 15.5 14.6 / 14.5 1.8 / 1.4 5.1 / 5.0 ROBERTA (us) 17.4 / 18.0 18.2 / 18.0 2.4 / 2.6 5.4 / 5.4 ETC (us) 18.3 / 18.8 16.1 / 17.2 2.4 / 2.7 5.4 / 5.4 READTWICE ( E ) 22.7 / 23.3 21.1 / 21.1 3.6 / 4.0 6.7 / 7.0 Table 2: Results on the NarrativeQA's development / test splits. Inter-segment memory matters We introduce a variant READTWICE -E(SS) (where SS stands for Single Segment) to isolate the gains from the memory layer. READTWICE -E(SS) prevents segments from attending to memories of other segments, thus disabling long-range dependency modeling. We observe that READTWICE-E improves over READTWICE -E(SS) on all tasks, modestly but non-negligibly for TQA, and significantly for HQA and especially NQA. This matches our knowledge of those datasets: TQA questions are based on a relatively short context and can typically be answered using a single passage in the context document. HQA questions have a similarly sized context, but are explicitly constructed to require information from multiple paragraphs to answer, and READTWICE shows accordingly larger gains. Finally, NQA has much larger contexts, and its questions generally require information from different parts of the document, increasing the importance of long-range dependency modeling and accordingly, the performance boost from READTWICE . Entities matter Entity mentions appears to be the most effective memory type in most experiments, leading to noticeably improved performance on both HQA and NQA. The difference is most pronounced in NQA whose particularly long and challenging contexts make it a perfect testbed. Source of non-memory gains The non-memory gains over a baseline ROBERTA model originate from the two extra layers and the entity-based MLM objective. In order to disentangle the sources of gains we train the READTWICE -E(SS) model using a 10-layer Transformer for BERT 1 (denoted as E ( SS , 10L ) in Table 3), with the same number of layers as ROBERTA . While the gains from 2 extra layers are significant ( E ( SS ) vs E ( SS , 10L ) ), most of the gains appear to result from the custom Model HQA NQA -R NQA -B TQA E 75.89 22.71 21.07 80.7 E ( SS ) 75.08 21.93 18.39 80.3 E ( SS , 10L ) 74.70 21.39 18.37 80.4 ROBERTA 72.00 17.40 18.2 75.9 CLS 75.32 20.89 17.80 80.6 STS 75.39 21.08 18.38 80.4 Table 3: Ablation studies on variants of READTWICE on the dev sets. READTWICE performs well on several QA tasks, particularly NarrativeQA where long-range dependencies among entities appear to be very important. The proposed method is conceptually simple, easy to implement and is capable of reading entire books. For future work, we plan to explore new memory types, hierarchies and aggregation functions. We also aim to apply the model to other tasks, particularly long text summarization, likely to benefit from a memory-forming mechanism. We thank Santiago Ontanon, Manzil Zaheer, Sudeep Gandhe, Anirudh Ravula, Bhargav Kana-gal, Jules Gagnon-Marchand and Sumit Sanghai for insightful discussions, Mou et al. (2020) for a sample evaluation code for NarrativeQA and reviewers for their feedback. This work is partially supported by NSF Awards IIS-1513966/ 1632803/1833137, CCF-1139148, DARPA Awards#: FA8750-18-2-0117, FA8750-19-1-0504, DARPA-D3M Award UCB-00009528, Google Research Awards, gifts from Facebook and Netflix, and ARO# W911NF-12-1-0241 and W911NF-15-1-0484.
abstain objective abstain objective abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain method abstain method abstain abstain method method abstain other other other other other other other other other other abstain other other method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain result other other abstain abstain abstain method abstain abstain abstain abstain abstain method other abstain abstain objective objective other other