Bounded-Memory Control Pappas   Dani Yogatama  Zhaofeng Wu  Schwartz  Noah A. Smith  Science & Engineering, University of Washington  Allen Institute for Artificial Intelligence Engineering, Hebrew University of Jerusalem Science, The University of Hong Kong {hapeng,jkasai,npappas,zfw7,nasmith}@cs.washington.edudyogatama@deepmind.com,lpk@cs.hku.hkroy.schwartz1@mail.huji.ac.il Abstract Transformer architectures have achieved state-of-the-art results on a variety of natural language processing (NLP) tasks. However, their attention mechanism comes with a quadratic complexity in sequence lengths, making the computational overhead prohibitive, especially for long sequences. Attention context can be seen as a random-access memory with each token taking a slot. Under this perspective, the memory size grows linearly with the sequence length, and so does the overhead of reading from it. One way to improve the efficiency is to bound the memory size. We show that disparate approaches can be subsumed into one abstraction, a ttention with b ounded-memory c ontrol (ABC ), and they vary in their organization of the memory. ABC reveals new, unexplored possibilities. First, it connects several efficient attention variants that would otherwise seem distinct. Second, this abstraction gives new insightsan established approach (Wang et al., 2020b) previously thought to not be applicable in causal attention, actually is . Last, we present a new instance of ABC , which draws inspiration from existing ABC approaches, but replaces their heuristic memory-organizing functions with a learned , contextualized one. Our experiments on language modeling, machine translation, and masked language model finetuning show that our approach outperforms previous efficient attention models; compared to strong transformer baselines, it significantly improves the inference time and space efficiency with no or negligible accuracy loss. Transformer architectures are now central in natural language processing (Vaswani et al., 2017). They rely on the attention mechanism (Bahdanau et al., 2015) to contextualize the input. The context can be seen as a random access memory whose size linearly grows with the sequence length; each query  This work was done while Zhaofeng Wu and Nikolaos Pappas were at the University of Washington. reads from it using a softmax-normalized linear combination, with overhead linear in the memory size. This amounts to a quadratic complexity overall, making transformers' computational overhead prohibitive, especially for long sequences. One way to improve attention's efficiency is to bound its memory size. Imposing a constant-sized constraint over the memory ensures that reading from it has constant time and space overhead, yielding a linear overall complexity in sequence lengths. This is in fact a common strategy adopted by several recent works. In this work, we show that some of these works are closely connected in ways that, to date, have gone unremarked. We propose a ttention with b ounded-memory c ontrol (ABC ), a unified abstraction over them. In ABC , constant-sized memories are organized with various control strategies, e.g., induced from heuristic patterns (Beltagy et al., 2020; Zaheer et al., 2020; Ainslie et al., 2020; Rae et al., 2020, inter alia ), locality assumptions (Parmar et al., 2018; Liu et al., 2018), or positions (Wang et al., 2020b). These strategies, by and large, are context-agnostic. In response to this, we propose ABCMLP , a particular instance of ABC that learns a contextualized control strategy from data. Specifically, ABCMLP uses a neural network to determine how to store each token into the memory (if at all). Compared to previous bounded-memory models, it strikes a better trade-off between accuracy and efficiency: controlling for the accuracy, ABCMLP can get away with much smaller memory sizes. ABC models (including ABCMLP ) come with a linear complexity in sequence lengths, and admit recurrent computation graphs in causal attention (self-attention over the prefix). Therefore they are appealing choices in a variety of applications, including text encoding, language modeling and text generation. This leads to a surprising finding. Linformer (Wang et al., 2020b), an established efficient attention method, was previously thought not 7469 to be applicable in causal attention or autoregressive decoding (Tay et al., 2020). Through the ABC view, we show that it actually is , and achieves competitive performance in our machine translation experiments. ABC connects existing models that would otherwise seem distinct, reveals new insights into established methods, and inspires new efficient attention architectures. We explore its applications in transformers, as a drop-in substitute for the canonical softmax attention. ABC offers a novel lens that can help future research in the analysis of transformers, where the theoretical insights are still catching up with empirical success. Experiments on language modeling, machine translation, and masked language model finetuning show that our ABCMLP model outperforms previous ABC approaches in accuracy with a much smaller memory size. Compared to the strong transformer baseline, ABCMLP achieves a significant speedup and memory savings at inference time, with no or negligible accuracy loss. The efficiency improvements are more prominent for long sequences, suggesting that the asymptotic savings are even more appealing in applications involving long sequences. We release our code at https://github.com/Noahs-ARK/ABC . This section presents our outer-product memory perspective of attention, which allows for a smooth transition to later discussion. In attention, a sequence of queries { q i } Ni =1 attend to a memory with N slots, each storing a key and value pair: K = [ k 1 , . . . , k N ]  , V = [ v 1 , . . . , v N ]   RN  d . 1 Query q reads from the memory using a softmax-normalized linear combination, producing a d -dimensional vector: attn( q , { k i } , { v i } ) = V  softmax ( Kq ) . This takes O ( N ) time and space. When the attention with N queries can be parallelized (e.g., in text encoding), it takes linear time and quadratic space; when it cannot be (e.g., in decoding), it takes quadratic time and linear space. The memory can be equivalently represented as sums of vector outer products: K = IK = (cid:80) Ni =1 e i  k i , V = (cid:80) Ni =1 e i  v i . I is the identity matrix, and  denotes the outer product: [ x  1 The number of queries and key-value pairs may differ, e.g., in the cross attention of a sequence-to-sequence model. y ] i,j = x i y j . N -dimensional vectors { e i } form the standard basis: e i has the i th element being one and others zeros. We can view e i as control vectors that determine where to store k i and v i : e i  k i = (cid:2) 0 , . . . 0 (cid:124) (cid:123)(cid:122) (cid:125) i  1 , 1 , 0 , . . . , 0 (cid:124) (cid:123)(cid:122) (cid:125) N  i (cid:3)   k i = (cid:2) 0 (cid:124)(cid:123)(cid:122)(cid:125) d  ( i  1) ; k i ; 0 (cid:124)(cid:123)(cid:122)(cid:125) d  ( N  i ) (cid:3)  . The N -byd matrix on the last line has its i th row being k  i and all others zeros; in this sense, k i is stored in the i th slot by e i , not affecting others. A straightforward way to improve attention's efficiency is to bound its memory size. Our outer-product view of attention provides a straightforward way to devise this, by replacing { e i } with control vectors that select n  N vectors to attend to. We dub this approach a ttention with b ounded-memory c ontrol (ABC ). Concretely, let (cid:101) K , (cid:101) V  R n  d denote a constant-size memory with n slots, with n set a priori . {  i  R n } Ni =1 denotes a sequence of control vectors. The output is calculated by attending to (cid:101) K and (cid:101) V : ABC ( q , { k i } , { v i } , {  i } ) = (cid:101) V  softmax (cid:16) (cid:101) Kq (cid:17) . (4) We will discuss various ways to construct {  i } in the subsequent sections. Reading from the memory takes a constant O ( n ) time and space; therefore ABC 's overall complexity is O ( Nn ) , linear in the sequence length. 2 Eq. 3 offers an equivalent recurrent computation, which is particularly useful in causal attention where only the prefix is looked at, (cid:101) K t +1 = (cid:101) K t +  t +1  k t +1 , (5) likewise for (cid:101) V t . In what follows, we study several existing efficient attention approaches and show that they are in fact instances of the ABC abstraction. Linformer (Wang et al., 2020b) is an established efficient transformer variant that has proven successful in masked language modeling and text encoding. It assumes fixed-length inputs and learns a low-rank approximation of the attention weights. A learned n -byN matrix WLF down projects the N -byd dimensional keys and values along the timestep dimension, to an n -byd memory: (cid:101) KLF = WLFK , (cid:101) VLF = WLFV ; they are then used for attention computation with Eq. 4. This yields a linear complexity in the input length. Linformer is an ABC instance with  LF i = WLF : ,i ( i th column), and in this sense, it learns a control vector for each position. Previous works have noted that Linformer cannot be efficiently applied in causal attention (Table 1 of Tay et al., 2020). Indeed, it is less straightforward to avoid mixing future with the past when projecting along the timestep dimension. ABC reveals that, in fact, Linformer is applicable in causal attention. Like all ABC models, it admits a linear-complexity recurrent computation (Eq. 5): (cid:101) KLF t +1 = (cid:101) K t +  LF t +1  k t +1 . This confirms ABC 's benefits: it reveals new insights about existing models and reassesses their applications and impact. Our experiments show that Linformer achieves competitive performance in machine translation. Improving attention's efficiency with clustering has received an increasing amount of interest (Kitaev et al., 2020; Roy et al., 2020; Wang et al., 2020a, inter alia ). ABC bears interesting connections to clustering-based methods. Here we discuss an approach that closely follows Vyas et al. (2020), except that it clusters keys and values instead of queries, and only attends to the centroids to reduce the effective context size. Formally, keys and values are grouped into n < N clusters { (cid:101) k CL j } nj =1 , { (cid:101) v CL j } nj =1 . 3 Let an N -byn binary matrix M denote the cluster membership shared between keys and values. M i,j = 1 iff. k i is assigned to cluster (cid:101) k CL j and v i to (cid:101) v CL j . The j th centroid for the keys is (cid:101) k CL j = N (cid:88) i =1 M i,j (cid:80) N =1 M ,j k i ; (6) 3 We use (cid:101) k CL j to denote both the j th cluster and its centroid. The last line indicates that this model is an instance of ABC :  i = (cid:80) nj =1 ( M i,j / (cid:80) N =1 M ,j ) e j . The stack of centroids can be seen as the constant-size memory. Putting aside the clustering overhead (i.e., constructing M and computing centroids), it has a linear complexity in the sequence length. In some applications, being able to remove entries from the memory can be beneficial: clearing up older context frees slots for more recent ones, promoting a locality inductive bias. ABC offers the capability to do so, if augmented with an additional matrix multiplication. We use the sliding-window attention as an example. Attending to the most recent n input tokens (Belt-agy et al., 2020; Zaheer et al., 2020; Sukhbaatar et al., 2021, inter alia ) can be seen as a first-in-first-out queue that pops out the oldest token while pushing in the most recent one: (cid:101) KWD t = [ k t  n +1 , ..., k t ]  . The pop operation can be achieved by multiplying an n -byn upper shift matrix : U i,j =  i +1 ,j , with  being the Kronecker delta (i.e., U has ones only on the superdiagonal and zeros elsewhere). Left-multiplying U against (cid:101) KWD t shifts its rows one position up, with zeros appearing in the last: U (cid:101) KWD t = U (cid:2) k t  n +1 , . . . , k t (cid:124) (cid:123)(cid:122) (cid:125) n (cid:3)  = (cid:2) k t  n +2 , . . . , k t  1 , k t (cid:124) (cid:123)(cid:122) (cid:125) n  1 , 0 (cid:3)   R n  d . Then the most recent token can be put into the slot freed up: (cid:101) KWD t +1 = U (cid:101) KWD t + e n  k t +1 . U and  t = e n ensure a first-in-first-out queue. Dilated and stride convolution patterns (Beltagy et al., 2020) can be similarly recovered (A.4). Recurrently multiplying U simulates the discrete pop operation (Grefenstette et al., 2015; Joulin and Mikolov, 2015; Yogatama et al., 2018) in a differentiable way. This is reminiscent of recurrent neural networks, while in this case U is never updated as 7471 parameters. It is exciting to explore learning U , but is beyond the scope of this work. Discussion. Besides the models discussed above, certain variants of Rae et al. (2020) and sparse attention patterns (local-to-global attention; Beltagy et al., 2020; Zaheer et al., 2020; Ainslie et al., 2020) can also be seen as instances of ABC (A). ABC provides a unified perspective of them, and at the same time points out their limitations: their control strategies are context-agnostic. In response to this, in 4 we propose to learn a contextualized strategy from data. Table 1 analyzes various ABC models, and Table 2 details their complexity. The ABC abstraction connects several existing approaches that would otherwise seem distinct. This inspires the design of new architectures. We hypothesize that learning a contextualized strategy can achieve better performance. This section introduces ABCMLP . It parameterizes  with a single-layer multi-layer perceptron (MLP) that takes as input the token's representation x i , and determines which slots to write it into and how much. Matrix W  is learned. exp is an elementwise activation function. The motivation is to allow for storing a fractional (but never negative) amount of input into the memory. 4 Using a non-negative activation, however, has a drawback: the scales of (cid:80) i  i  k i and (cid:80) i  i  v i would grow with the sequence lengths, making training less stable. To overcome this, we divide  i vectors by their sum. This functions as normalization and aims to offset the impact of varying sequence lengths. 5 It admits the recurrent computation graph as in Eq. 5, and has a linear complexity in the sequence length. A key design choice of ABCMLP is that its  i depends only on current input x i . This helps (1) keep the recurrent computation efficient in practice (Lei et al., 2018), and (2) make it applicable 4 We experiment with other activations in C.2. 5 Here encoder self-attention or cross attention is assumed, and the normalization sums over the entire sequence. Causal attention is slightly different, normalizing by the sum over the prefix instead:  i =  i / (cid:80) ij =1  j . This does not require access to future tokens. B.1 details a linear complexity computation graph of causal  i . in not only encoder self-attention and cross attention, but also causal attention. Concurrently to this work, Goyal et al. (2021) and Ma et al. (2021) also proposed methods to learn contextualized control. They compute  i from previous layer's memory, revealing the full sequence to the control vectors. As a result, these two approaches are unsuitable for causal attention. 6 ABCMLP , as other ABC models, can be used as a drop-in replacement for the canonical softmax attention, and we apply its multihead variant in transformers. With proper parameter sharing, the number of additional parameters ABCMLP incurs is small: inspired by Wang et al. (2020b), we tie  -MLP's parameters across different layers, which adds less than 1% parameters to the models. ABCMLP : context-agnostic then context-dependent attention. We now dissect ABCMLP and show that it can be seen as a cascade of two attention mechanisms: one with a learned context-agnostic pseudo query followed by one with a context-dependent query. Our analysis starts with a one-dimensional example; the conclusion generalizes to higher-dimensional cases. Example 1. Consider ABCMLP with a single memory slot ( n = 1 ). It is parameterized with a learned vector w  , and  i = exp( w   x i ) / (cid:80) Nj =1 exp( w   x j ) . Since  i is a scalar here,  i  k i =  i k  i . In other words, (cid:101) K uses w  as a pseudo-query to attend to { x i } and { k i } . Likewise, (cid:101) V  = attn( w  , { x i } Ni =1 , { v i } Ni =1 ) . Despite its similarity to the standard softmax attention, Example 1 has a more efficient linear complexity in sequence lengths. w  's being context-independent is the key to the savings. Table 2 details its complexity. Example 1's conclusion generalizes to higher-dimensional cases: the j th dimension of {  i } attends to { x i } and { k i } using the j th row of W  as the context-independent pseudo-query; n such attention mechanisms run in parallel, stacking the 6 Both are instances of ABC (A.5). results into n -byd memory (cid:101) K and (cid:101) V . Intuitively, it is the real queries { q i } that encode what information is useful for the prediction task. Without access to them, ABCMLP summarizes the input for n times using different pseudo-queries, aiming to preserve enough information in the memory for onward computation. The attention output is calculated with the context-dependent real queries using Eq. 4. B.2 presents a detailed derivation. starting from distinct motivations, ABCMLP closely relates to hierarchical attention (HA; Yang et al., 2016). HA summarizes the context into higher-level representations with a cascade of attention mechanisms, e.g., words to sentences, and then to documents. ABCMLP applies two types of attention. The first learns context-agnostic pseudo-queries and attends to the same sequence for n times in parallel, while the second retrieves from the memory with real queries. HA, in contrast, summarizes non-overlapping segments at each level. The learned pseudo-queries closely relate to the inducing point method in set attention (ISA; Lee et al., 2019). ISA applies a non-linear feedforward network between a cascade of two attention modules. This precludes the outer-product memory computation and efficient recurrences in ABC . Another line of work linearizes attention through kernel tricks and also applies bounded memory: their feature map dimensions are analogous to memory sizes. They substitute the softmax with approximations (Peng et al., 2021; Choromanski et al., 2021), heuristically designed (Katharopoulos et al., 2020; Schlag et al., 2021), or learned (Kasai et al., 2021b) functions. ABCMLP keeps the softmax, but over a smaller constant-sized context. This can be useful in practice: (1) ABC provides a unified perspective of several efficient attention methods, allowing for borrowing from existing wisdom to design new architectures; (2) it draws a close analogy to the canonical softmax attention, and is better-suited as its drop-in substitute in various application settings, as we will show in the experiments; (3) empirically, we find that ABCMLP can get away with a much smaller memory size to retain the accuracy. Peng et al. (2021) and Schlag et al. (2021) use gating to promote recency bias. The same technique is equally applicable in ABC models. ral Turing machines (NTM; Graves et al., 2014). ABCMLP computes the control vectors {  i } as a function of the input, but not of the memory as in NTM. This ensures that the control vectors at different timesteps can be computed in parallel, improving the time efficiency in practice (Lei et al., 2018; Peng et al., 2018). Analogies between memory and neural architectures are also made by other previous works (Hochreiter and Schmidhuber, 1997; Weston et al., 2015; Le et al., 2020, inter alia ). We evaluate ABC models on language modeling (5.1), sentence-level and document-level machine translation (5.2), and masked language model finetuning (5.3). Dataset statistics and implementation details are summarized in C. Setting. We experiment with WikiText-103, sampled text from English Wikipedia (Merity et al., 2017). The BASE model with standard softmax attention is the strong transformer-based language model by Baevski and Auli (2019). We compare the following ABC variants, which build on BASE , but replace the softmax attention with linear-complexity bounded-memory attention alternatives while keeping other components the same. ABCMLP , as described in 4, learns a contextualized exp -MLP as the  function. Linformer (3.1; Wang et al., 2020b). ABCRD stores each token in a randomly-selected memory slot with  t = e i t . i t is uniformly drawn from { 1 , . . . , n } at each time step. This helps us quantify the differences between random and learned bounded-memory controls. We consider two model size settings:  16 layers (Baevski and Auli, 2019). All models have around  242M parameters. They train with 512-token segments, and evaluate with 0 or 480 context sizes: a 0or 480length prefix precedes each evaluation segment. 32 layers (Kasai et al., 2021b). All models have  484M parameters. This setting applies layer dropout (Fan et al., 2020), and evaluates with a 256 context size. It aims to compare ABCMLP to several kernel-based efficient attention variants: ELU (Katharopoulos et al., 2020), RFA (Peng et al., 2021), and T2R (Kasai et al., 2021b). (a) 16-layer setting. 0/480 indicate evaluation context sizes. (b) 32-layer setting. A 256-length context is used at evaluation time. numbers are due to Kasai et al. (2021b). ABC models, ABCMLP achieves the best performance for both context sizes. With a memory size n = 64 , ABCMLP outperforms both Linformer and ABCRD by more than 2.9 test perplexity; and the gap is larger with the longer 480-length context: more than 3.6 test perplexity. ABCMLP -32 outperforms its larger-memory ABC counterparts by more than 2.1 test perplexity. These results confirm ABCMLP 's advantages of using a contextualized strategy. Surprisingly, Linformer underperforms ABCRD , and its performance drops with the larger 480-length context window. This suggests that, while successful in text encoding, Lin-former's position-based strategy is a suboptimal design choice for causal attention, at least for long context. All ABC models underperform the BASE , with ABCMLP -64 having the smallest gap of 0 . 5 perplexity. ABCMLP -32 outperforms kernel-based methods by more than 0.9 test perplexity, using Kasai et al. (2021b)'s 32-layer setting (Table 3b). Datasets. To assess their performance over various output lengths, we compare ABC models on sentenceand documentlevel machine translation. Sentence-level translation with WMT14 EN-DE 7474 Model Cross n Causal n BLEU BASE-27.2 ABCRD 32 32 25.7 ABCRD 64 64 26.2 Linformer 32 32 26.6 Linformer 64 64 26.7 ABCMLP 32 8 27.1 ABCMLP 32 32 27.3 (a) Bolded number outperforms BASE . seeds. Bold number performs the best among ABC models. (Bojar et al., 2014). The preprocessing and data splits follow Vaswani et al. (2017). Document-level translation with IWSLT14 ES-EN (Cettolo et al., 2014). We use Miculicich et al. (2018)'s data splits and preprocessing. Following standard practice (Voita et al., 2019), a 4-sentence sliding window is used to create the dataset, i.e., each instance has 4 sentences. Setting. We compare ABC variants as in 5.1. C.2 further compares to the clustering-based (3.2) and sliding-window (3.3) ABC variants. The BASE model they build on is our implementation of transformer-base (Vaswani et al., 2017). ABC variants replace decoder cross attention and causal attention with bounded-memory attention, while keeping softmax attention for the encoder, since its overhead is much less significant (Kasai et al., 2021a); other components are kept the same. C.2 studies a model that replaces all softmax attention with ABCMLP . It performs on par with BASE , confirming ABCMLP 's broad applicability in various application scenarios. We evaluate with SacreBLEU (Post, 2018). Results. Table 4a summarizes sentence-level machine translation results on the WMT14 EN-DE test set. Overall ABCMLP performs on par with BASE , with either 32-32 cross-causal memory sizes or 32-8. Even with smaller memory sizes, it outperforms other ABC variants by more than 1.1 BLEU. Differently from the trend in the language modeling experiment (5.1), Linformer outperforms ABCRD by more than 0.5 BLEU. We attribute this to the smaller sequence lengths of this dataset. ABCMLP outperforms other ABC models by more than 0.4 BLEU, even with smaller memory sizes. The trend is similar on document-level translation with IWSLT14 ES-EN (Table 4b), except that ABCMLP slightly underperforms BASE by 0.2 BLEU. This suggests that even with longer sequences, ABCMLP is effective despite its bounded memory size. Linformer fails to converge even with multiple random seeds, suggesting the limitations of its purely position-based strategy in tasks involving decoding varying-length text. Setting. We compare the ABC variants as in 5.1. It is interesting to pretrain ABC from scratch, but we lack the resources to do so. Instead, we warm-start from a pretrained RoBERTa-base (Liu et al., 2019) trained with the softmax transformer, swap its attention with ABC variants, and continue pretraining with the masked language modeling (MLM) objective on a concatenation of BookCorpus (Zhu et al., 2015), English Wikipedia, OpenWebText (Gokaslan and Cohen, 2019), and RealNews (Zellers et al., 2019). 7 Then the models are finetuned and evaluated on downstream classification datasets from the the GLUE benchbark (Wang et al., 2019). This is an appealing setting, since it avoids reinvesting the huge amounts of resources already put into pretraining. 8 Results. Table 5 compares downstream text classification performance. BASE indicates a baseline that continues pretraining RoBERTa-base on our data. 9 Following standard practice, we report development accuracy. Linformer achieves competitive 7 Our data differs from RoBERTa's, which we do not have access to. We replace CC-News (Nagel, 2016) with RealNews, and drop Stories (Trinh and Le, 2018), whose public access is broken at the time of this work. 8 In preliminary experiments, we explored swapping in ABC , and then directly finetuning on downstream tasks without continued MLM pretraining; all models fail. 9 BASE slightly underperforms RoBERTa-base. This could be due to overfitting, or the pretraining data discrepancy. performance, aligned with Wang et al. (2020b)'s results. ABCMLP outperforms Linformer, and performs on par with or better than BASE , affirming the benefits of using contextualized memory organization in MLM. ABCRD fails to converge in continued pretraining even with multiple seeds. Based on the above results, we think ABCMLP can achieve competitive performance when pretrained from scratch, just as Linformer does (Wang et al., 2020b). Further empirical exploration is beyond our budget and left for future work. Decoding efficiency over varying sequence lengths. ABC 's efficiency gains can be more prominent for long sequences. We study ABCMLP 's decoding overhead with varying sequence lengths. Following Kasai et al. (2021b), we consider a sequence-to-sequence generation experiment. Three linear-complexity models are compared: RFA (with 256/128 cross/causal memory sizes; Peng et al., 2021), T2R (32/4; Kasai et al., 2021b), and ABCMLP (32/8). The sizes are chosen to maximize efficiency without accuracy drop. T2R needs to be finetuned from a pretrained transformer to match its performance, while others don't. All linear-time models achieve consistent decoding speed for different lengths (Figure 1a), substantially outpacing the softmax attention baseline, especially for long sequences. In particular, ABCMLP decodes  1.25 times faster than RFA, another competitive model that can match trans-former's accuracy without a warm start from a pretrained model. This can be attributed to the fact that ABCMLP achieves similar accuracy with a much smaller memory. T2R's memory sizes are similar to ABCMLP 's, but it decodes about 20% faster. This is because it does not compute the softmax when calculating attention output, while ABCMLP does (Eq. 4). These results show that ABCMLP is an appealing modeling choice for decoding tasks, especially when training from scratch is desired. ABCMLP also achieves significant savings in terms of memory overhead (Figure 1b). ABCMLP , RFA, and T2R's curves are similar. Text encoding efficiency. We compare the efficiency of ABCMLP against softmax attention and Linformer when used as text encoders. The mod-els' sizes mirror those in the MLM experiment (5.3). Table 6 summarizes inference time and memory overhead with 512-length inputs, batch size 16. Both ABCMLP and Linformer achieve inference speed gains and memory savings over BASE . Linformer is faster, since its linear projection is cheaper to compute than ABCMLP 's MLP. Inference speed is measured on the same V100 GPU. The trend in memory overhead is similar. Although ABCMLP slightly underperforms Linformer in terms of inference speed, it can be a more appealing architectural choice in practice: in all of our 5 experiments, ABCMLP outperforms other ABC models in accuracy. Linformer, in contrast, fails to converge or yields sub-optimal performance on some tasks. This confirms its flexibility and ap-7476 BASE Linformer ABCMLP n 64 128 64 128 Speed 1.0  1.7  1.5  1.5  1.3  Memory 1.0  0.5  0.6  0.5  0.6  Table 6: Text encoding inference speed (higher is better) and memory (lower is better). Memory size's impact on accuracy. Practically, one may want to minimize the memory size to improve efficiency. We use the WMT14 EN-DE experiment to investigate how memory size affects accuracy. Using the 5.2's setup, we vary ABCMLP 's cross and causal attention memory sizes and compare their translation quality on the development data. They are selected from { 8 , 16 , 32 , 64 } , with cross attention's equal to or larger than causal's: cross attention is more important than causal attention in machine translation (Michel et al., 2019). Our results (Table 7) align with this observation: when cross attention memory is large enough, reducing causal attention memory size from 64 to 8 has a minor 0.3 BLEU drop. Surprisingly, ABCMLP with 8-8 sized cross-causal memory is only 1.1 BLEU behind the best-performing configuration. We presented attention with bounded-memory control (ABC ). It provides a unified perspective of several recently-proposed models, and shows that they vary in the organization of the bounded memory. ABC reveals new insights into established methods and inspires new architectures. We proposed ABCMLP , a particular instance of ABC that learns a contextualized memory control. On language modeling, machine translation, and masked language model finetuning, ABCMLP outperforms previous ABC models. Compared to the strong transformer baseline, ABCMLP achieves substantial efficiency improvements with no or negligible accuracy loss. We would like to thank the ARK group at the University of Washington for their helpful feedback, and the anonymous reviewers for their thoughtful comments. This work was supported in part by NSF grant 2113530 and a Google Fellowship. Nikolaos Pappas was supported by the Swiss National Science Foundation grant P400P2_183911.
abstain abstain abstain abstain abstain abstain result abstain abstain abstain objective result abstain abstain result abstain abstain abstain abstain abstain result objective other abstain objective abstain abstain abstain abstain abstain abstain result abstain objective abstain result abstain abstain other method abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain objective abstain abstain other other other