Transformers are unable to model long-term memories effectively, since the amount of computation they need to perform grows with the context length. While variations of efficient transformers have been proposed, they all have a finite memory capacity and are forced to drop old information. In this paper, we propose the  -former , which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space attention mechanism to attend over the long-term memory, the  -former's attention complexity becomes independent of the context length, trading off memory length with precision. In order to control where precision is more important,  -former maintains sticky memories, being able to model arbitrarily long contexts while keeping the computation budget fixed. Experiments on a synthetic sorting task, language modeling, and document grounded dialogue generation demonstrate the  -former's ability to retain information from long sequences. 1 1 Introduction When reading or writing a document, it is important to keep in memory the information previously read or written. Humans have a remarkable ability to remember long-term context, keeping in memory the relevant details (Carroll, 2007; Kuhbandner, 2020). Recently, transformer-based language models have achieved impressive results by increasing the context size (Radford et al., 2018, 2019; Dai et al., 2019; Rae et al., 2019; Brown et al., 2020). However, while humans process information sequentially, updating their memories continuously, and recurrent neural networks (RNNs) update a single memory vector during time, transformers do not  they exhaustively query every representation associated to the past events. Thus, the amount 1 The code is available at https://github.com/ deep-spin/infinite-former . of computation they need to perform grows with the length of the context, and, consequently, transformers have computational limitations about how much information can fit into memory. For example, a vanilla transformer requires quadratic time to process an input sequence and linear time to attend to the context when generating every new word. Several variations have been proposed to address this problem (Tay et al., 2020b). Some propose using sparse attention mechanisms, either with data-dependent patterns (Kitaev et al., 2020; Vyas et al., 2020; Tay et al., 2020a; Roy et al., 2021; Wang et al., 2021) or data-independent patterns (Child et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020), reducing the self-attention complexity (Katharopoulos et al., 2020; Choromanski et al., 2021; Peng et al., 2021; Jaegle et al., 2021), and caching past representations in a memory (Dai et al., 2019; Rae et al., 2019). These models are able to reduce the attention complexity, and, consequently, to scale up to longer contexts. However, as their complexity still depends on the context length, they cannot deal with unbounded context. In this paper, we propose the  -former ( infinite former ; Fig. 1): a transformer model extended with an unbounded long-term memory (LTM), which allows the model to attend to arbitrarily long contexts. The key for making the LTM unbounded is a continuous-space attention framework (Mar-tins et al., 2020) which trades off the number of information units that fit into memory (basis functions) with the granularity of their representations. In this framework, the input sequence is represented as a continuous signal , expressed as a linear combination of N radial basis functions (RBFs). By doing this, the  -former's attention complexity is O ( L 2 + L  N ) while the vanilla transformer's is O ( L  ( L + LLTM )) , where L and LLTM correspond to the transformer input size and the long-term memory length, respectively (details in 3.1.1). Thus, this representation comes with 5468 two significant advantages: (i) the context can be represented using a number of basis functions N smaller than the number of tokens, reducing the attention computational cost; and (ii) N can be fixed , making it possible to represent unbounded context in memory, as described in 3.2 and Fig. 2, without increasing its attention complexity. The price, of course, is a loss in resolution: using a smaller number of basis functions leads to lower precision when representing the input sequence as a continuous signal, as shown in Fig. 3. To mitigate the problem of losing resolution, we introduce the concept of sticky memories (3.3), in which we attribute a larger space in the LTM's signal to regions of the memory more frequently accessed. This creates a notion of permanence in the LTM, allowing the model to better capture long contexts without losing the relevant information, which takes inspiration from long-term potentiation and plasticity in the brain (Mills et al., 2014; Bamji, 2005). To sum up, our contributions are the following:  We propose the  -former, in which we extend the transformer model with a continuous long-term memory (3.1). Since the attention computational complexity is independent of the context length, the  -former is able to model long contexts. We propose a procedure that allows the model to keep unbounded context in memory (3.2). We introduce sticky memories, a procedure that enforces the persistence of important information in the LTM (3.3). We perform empirical comparisons in a synthetic task (4.1), which considers increasingly long sequences, in language modeling (4.2), and in document grounded dialogue generation (4.3). These experiments show the benefits of using an unbounded memory. A transformer (Vaswani et al., 2017) is composed of several layers, which encompass a multi-head self-attention layer followed by a feed-forward layer, along with residual connections (He et al., 2016) and layer normalization (Ba et al., 2016). input size and e is the embedding size of the attention layer. The queries Q , keys K , and values V , to be used in the multi-head self-attention computation are obtained by linearly projecting the input, or the output of the previous layer, X , for each attention head h : Q h = X h WQ h , K h = X h WK h , V h = X h WV h , (1) where WQ h , WK h , WV h  R d  d are learnable projection matrices, d = e / H , and H is the number of heads. Then, the context representation Z h  RL  d , that corresponds to each attention head h , is obtained as: Z h = softmax where the softmax is performed row-wise. The head context representations are concatenated to obtain the final context representation Z  RL  e : Z = [ Z 1 , . . . , ZH ] WR , Continuous attention mechanisms (Martins et al., 2020) have been proposed to handle arbitrary continuous signals, where the attention probability mass function over words is replaced by a probability density over a signal. This allows time intervals or compact segments to be selected. To perform continuous attention, the first step is to transform the discrete text sequence represented by X  RL  e into a continuous signal. This is done by expressing it as a linear combination of basis functions. To do so, each x i , with i  { 1 , . . . , L } , is first associated with a position in an interval, t i  [0 , 1] , e.g. , by setting t i = i/L . Then, we obtain a continuous-space representation  X ( t ) R e , for any t  [0 , 1] as:  X ( t ) = B (cid:62) ( t ) , where ( t ) RN is a vector of N RBFs, e.g.,  j ( t ) = N ( t ;  j ,  j ) , with  j  [0 , 1] , and B  RN  e is a coefficient matrix. B is obtained with multivariate ridge regression (Brown et al., 1980) so that  X ( t i ) x i for each i  [ L ] , which leads to the closed form (see App. A for details): B where F = [ ( t 1 ) , . . . , ( t L )]  RN  L packs the basis vectors for the L locations. As G  RL  N only depends of F , it can be computed offline. Having converted the input sequence into a continuous signal  X ( t ) , the second step is to attend over this signal. To do so, instead of having a discrete probability distribution over the input sequence as in standard attention mechanisms (like in Eq. 2), we have a probability density p , which can be a Gaussian N ( t ; ,  2 ) , where  and  2 are computed by a neural component. A unimodal Gaussian distribution encourages each attention head to attend to a single region, as opposed to scattering its attention through many places, and enables tractable computation. Finally, having p , we can compute the context vector c as: c = E p Martins et al. (2020) introduced the continuous attention framework for RNNs. In the following section (3.1), we will explain how it can be used for transformer multi-head attention. To allow the model to access long-range context, we propose extending the vanilla transformer with a continuous LTM, which stores the input embeddings and hidden states of the previous steps. We also consider the possibility of having two memories: the LTM and a short-term memory (STM), which consists in an extension of the transformer's hidden states and is attended to by the transformer's self-attention, as in the transformer-XL (Dai et al., 2019). A diagram of the model is shown in Fig. 1. 3.1 Long-term Memory For simplicity, let us first assume that the long-term memory contains an explicit input discrete sequence X that consists of the past text sequence's input embeddings or hidden states, 2 depending on the layer 3 (we will later extend this idea to an unbounded memory in 3.2). where ( t ) RN are basis functions and coefficients B  RN  e are computed as in Eq. 5, 2 We stop the gradient with respect to the word embeddings or hidden states before storing them in the LTM. B (cid:62) = X (cid:62) G . Then, we can compute the LTM keys, K h  RN  d , and values, V h  RN  d , for each attention head h , as: K h = B h WK h , V h = B h WV h , (8) where WK h , WV h  R d  d are learnable projection matrices. 4 For each query q h,i for i  { 1 , . . . , L } , we use a parameterized network, which takes as input the attention scores, to compute  h,i  ]0 , 1[ and  2 h,i  R > 0 :  h,i =sigmoid density h,i as ( t ;  h,i ,  2 h,i ) . Finally, having the value function  V h ( t ) given as  V h ( t ) = V (cid:62) h ( t ) , we compute the head-specific representation vectors as in Eq. 6: z h,i = E p h,i [  V h ] = V h E p h,i [ ( t )] (11) which form the rows of matrix ZLTM ,h  RL  d that goes through an affine transformation, ZLTM = [ ZLTM , 1 , . . . , ZLTM ,H ] WO . (cid:62) The long-term representation, ZLTM , is then summed to the transformer context vector, ZT , to obtain the final context representation Z  RL  e : Z = ZT + ZLTM , (12) which will be the input to the feed-forward layer. As the  -former makes use of a continuous-space attention framework (Martins et al., 2020) to attend over the LTM signal, its key matrix size K h  RN  d depends only on the number of basis functions N , but not on the length of the context being attended to. Thus, the  -former's attention complexity is also independent of the context's length. It corresponds to O ( L ( L + LSTM ) + L  N ) when also using a short-term memory and O ( L 2 + L  N ) when only using the LTM, both (cid:28) O ( L ( L + LLTM )) , which would be the complexity of a vanilla transformer attending to the same context. For this reason, the  -former can attend to arbitrarily long contexts without increasing the amount of computation needed. When representing the memory as a discrete sequence, to extend it, we need to store the new hidden states in memory. In a vanilla transformer, this is not feasible for long contexts due to the high memory requirements. However, the  -former can attend to unbounded context without increasing memory requirements by using continuous attention, as next described and shown in Fig. 2. To be able to build an unbounded representation, we first sample M locations in [0 , 1] and evaluate  X ( t ) at those locations. These locations can simply be linearly spaced, or sampled according to the region importance, as described in 3.3. Then, we concatenate the corresponding vectors with the new vectors coming from the short-term memory. For that, we first need to contract this function by a factor of   ]0 , 1[ to make room for the new vectors. We do this by defining: X contracted ( t ) = X ( t/ ) = B (cid:62) ( t/ ) . (13) Then, we can evaluate  X ( t ) at the M locations 0  t 1 , t 2 , . . . , t M   as: x m = B (cid:62) ( t m / ) , for m  [ M ] , (14) and define a matrix X past = [ x 1 , x 2 , . . . , x M ] (cid:62) RM  e with these vectors as rows. After that, we concatenate this matrix with the new vectors X new , obtaining: X = (cid:104) X past (cid:62) , X new (cid:62) (cid:105) (cid:62) R ( M + L ) e . (15) Finally, we simply need to perform multivariate ridge regression to compute the new coefficient matrix B  RN  e , via B (cid:62) = X (cid:62) G , as in Eq. 5. To do this, we need to associate the vectors in X past with positions in [0 ,  ] and in X new with positions in ] , 1] so that we obtain the matrix G  R ( M + L ) N . We consider the vectors positions to be linearly spaced. When extending the LTM, we evaluate its current signal at M locations in [0 , 1] , as shown in Eq. 14. These locations can be linearly spaced. However, some regions of the signal can be more relevant than others, and should consequently be given a larger memory space in the next step LTM's signal. To take this into account, we propose sampling the M locations according to the signal's relevance at each region (see Fig. 6 in App. B). To do so, we construct a histogram based on the attention given to each interval of the signal on the previous step. For that, we first divide the signal into 5471 D linearly spaced bins { d 1 , . . . , d D } . Then, we compute the probability given to each bin, p ( d j ) for j  { 1 , . . . , D } , as: p where H is the number of attention heads and L is the sequence length. Note that Eq. 16's integral can be evaluated efficiently using the erf function: a N ( t ; ,  2 ) = 1 2 erf b  2 erf a  2 . (17) Then, we sample the M locations at which the LTM's signal is evaluated at, according to p . By doing so, we evaluate the LTM's signal at the regions which were considered more relevant by the previous step's attention, and will, consequently attribute a larger space of the new LTM's signal to the memories stored in those regions. Discrete sequences can be highly irregular and, consequently, difficult to convert into a continuous signal through regression. Because of this, before applying multivariate ridge regression to convert the discrete sequence X into a continuous signal, we use a simple convolutional layer (with stride = 1 and width = 3 ) as a gate, to smooth the sequence:  X = sigmoid To train the model we use the cross entropy loss. Having a sequence of text X of length L as input, a language model outputs a probability distribution of the next word p ( x t +1 | x t , . . . , x t  L ) . Given a corpus of T tokens, we train the model to minimize its negative log likelihood: LNLL =  T  1 Additionally, in order to avoid having uniform distributions over the LTM, we regularize the continuous attention given to the LTM, by minimizing the Kullback-Leibler (KL) divergence, DKL , between the attention probability density, N (  h ,  h ) , and a Gaussian prior, N (  0 ,  0 ) . As different heads can attend to different regions, we set  0 =  h , regularizing only the attention variance, and get: LKL = T  1 (cid:88) t =0 H (cid:88) h =1 DKL ( N (  h ,  h ) || N (  h ,  0 )) (20) = T  1 (cid:88) t =0 H (cid:88) h =1 1 2 (cid:18) 2 h  20  log (cid:18) h  0 (cid:19) 1 (cid:19) . (21) Thus, the final loss that is minimized corresponds to: L = LNLL +  KLLKL , To understand if the  -former is able to model long contexts, we first performed experiments on a synthetic task, which consists of sorting tokens by their frequencies in a long sequence (4.1). Then, we performed experiments on language modeling (4.2) and document grounded dialogue generation (4.3) by fine-tuning a pre-trained language model. 5 4.1 Sorting In this task, the input consists of a sequence of tokens sampled according to a token probability distribution (which is not known to the system). The goal is to generate the tokens in the decreasing order of their frequencies in the sequence. One example can be: 1 2 1 3 1 0 3 1 3 2 To understand if the long-term memory is being effectively used and the transformer is not only performing sorting by modeling the most recent tokens, we design the token probability distribution to change over time : namely, we set it as a mixture of two distributions, p = p 0 + (1   ) p 1 , where the mixture coefficient   [0 , 1] is progressively increased from 0 to 1 as the sequence is generated. The vocabulary has 20 tokens and we experiment with sequences of length 4,000, 8,000, and 16,000. Baselines. We consider the transformer-XL 6 (Dai et al., 2019) and the compressive transformer 7 (Rae et al., 2019) as baselines. The transformer-XL consists of a vanilla transformer (Vaswani et al., 2017) extended with a short-term memory which is composed of the hidden states of the previous steps. The compressive transformer is an extension of the transformer-XL: besides the short-term memory, it has a compressive long-term memory which is composed of the old vectors of the short-term memory, compressed using a CNN. Both the transformer-XL and the compressive transformer require relative positional encodings. In contrast, there is no need for positional encodings in the memory in our approach since the memory vectors represent basis coefficients in a predefined continuous space. For all models we used a transformer with 3 layers and 6 attention heads, input size L = 1 , 024 and memory size 2,048. For the compressive transformer, both memories have size 1,024. For the  -former, we also consider a STM of size 1,024 and a LTM with N = 1 , 024 basis functions, having the models the same computational cost. Further details are described in App. C.1. Results. As can be seen in the left plot of Fig. 3, the transformer-XL achieves a slightly higher accuracy than the compressive transformer and  -former for a short sequence length (4,000). This is because the transformer-XL is able to keep almost the entire sequence in memory. However, its accuracy degrades rapidly when the sequence length is increased. Both the compressive trans-6 We use the authors' implementation available at https: //github.com/kimiyoung/transformer-xl . former and  -former also lead to smaller accuracies when increasing the sequence length, as expected. However, this decrease is not so significant for the  -former, which indicates that it is better at modeling long sequences. Regression error analysis. To better understand the trade-off between the  -former's memory precision and its computational efficiency, we analyze how its regression error and sorting accuracy vary when varying the number of basis functions used, on the sorting task with input sequences of length 8,000. As can be seen in the right plot of Fig. 3, the sorting accuracy is negatively correlated with the regression error, which is positively correlated with the number of basis functions. It can also be observed, that when increasing substantially the number of basis functions the regression error reaches a plateau and the accuracy starts to drop. We posit that the latter is caused by the model having a harder task at selecting the locations it should attend to. This shows that, as expected, when increasing  -former's efficiency or increasing the size of context being modeled, the memory loses precision. To understand if long-term memories can be used to extend a pre-trained language model, we fine-tune GPT-2 small (Radford et al., 2019) on Wikitext-103 (Merity et al., 2017) and a subset of PG-19 (Rae et al., 2019) containing the first 2,000 books (  200 million tokens) of the training set. To do so, we extend GPT-2 with a continuous long-term memory (  -former) and a compressed memory (compressive transformer) with a positional bias, 5473 Wikitext-103 PG19 GPT-2 16.85 33.44 Compressive 16.87 33.09  -former 16.64 32.61  -former based on Press et al. (2021). 8 For these experiments, we consider transformers with input size L = 512 , for the compressive transformer we use a compressed memory of size 512, and for the  -former we consider a LTM with N = 512 Gaussian RBFs and a memory threshold of 2,048 tokens, having the same computational budget for the two models. Further details and hyperparameters are described in App. C.2. Results. The results reported in Table 1 show that the  -former leads to perplexity improvements on both Wikitext-103 and PG19, while the compressive transformer only has a slight improvement on the latter. The improvements obtained by the  -former are larger on the PG19 dataset, which can be justified by the nature of the datasets: books have more long range dependencies than Wikipedia articles (Rae et al., 2019). In document grounded dialogue generation, besides the dialogue history, models have access to a document concerning the conversation's topic. In the CMU Document Grounded Conversation dataset (CMU-DoG) (Zhou et al., 2018), the dialogues are about movies and a summary of the movie is given as the auxiliary document; the auxiliary document is divided into parts that should be considered for the different utterances of the dialogue. In this paper, to evaluate the usefulness of the long-term memories, we make this task slightly more challenging by only giving the models access to the document before the start of the dialogue. We fine-tune GPT-2 small (Radford et al., 2019) using an approach based on Wolf et al. (2019). To allow the model to keep the whole document on memory, we extend GPT-2 with a continuous LTM (  -former) with N = 512 basis functions. As baselines, we use GPT-2, with and without ac-8 The compressive transformer requires relative positional encodings. When using only GPT-2's absolute positional encodings the model gives too much attention to the compressed memory, and, consequently, diverges. Thus, we adapted it by using positional biases on the attention mechanism. cess (GPT-2 w/o doc) to the auxiliary document, with input size L = 512 , and GPT-2 with a compressed memory with attention positional biases (compressive), of size 512. Further details and hyper-parameters are stated in App. C.3. To evaluate the models we use the metrics: perplexity, F1 score, Rouge-1 and Rouge-L (Lin, 2004), and Meteor (Banerjee and Lavie, 2005). Results. As shown in Table 2, by keeping the whole auxiliary document in memory, the  -former and the compressive transformer are able to generate better utterances, according to all metrics. While the compressive and  -former achieve essentially the same perplexity in this task, the  -former achieves consistently better scores on all other metrics. Also, using sticky memories leads to slightly better results on those metrics, which suggests that attributing a larger space in the LTM to the most relevant tokens can be beneficial. Analysis. In Fig. 4, we show examples of utterances generated by  -former along with the excerpts from the LTM that receive higher attention throughout the utterances' generation. In these examples, we can clearly see that these excerpts are highly pertinent to the answers being generated. Also, in Fig. 5, we can see that the phrases which are attributed larger spaces in the LTM, when using sticky memories, are relevant to the conversations. Continuous attention. Martins et al. (2020) introduced 1D and 2D continuous attention, using Gaussians and truncated parabolas as densities. They applied it to RNN-based document classi-fication, machine translation, and visual question answering. Several other works have also proposed the use of (discretized) Gaussian attention for natural language processing tasks: Guo et al. (2019) proposed a Gaussian prior to the self-attention mechanism to bias the model to give higher attention to nearby words, and applied it to natural language inference; You et al. (2020) proposed the use 5474 Cast: Macaulay Culkin as Kevin. Joe Pesci as Harry. Daniel Stern as Marv. John Heard as Peter. Roberts Blossom as Marley. The film stars Macaulay Culkin as Kevin McCallister, a boy who is mistakenly left behind when his family flies to Paris for their Christmas vacation. Kevin initially relishes being home alone, but soon has to contend with two would-be burglars played by Joe Pesci and Daniel Stern. The film also features Catherine O'Hara and John Heard as Kevin's parents. Previous utterance: Or maybe rent, anything is reason to celebrate..I would like to talk about a movie called "Home Alone" Answer: Macaulay Culkin is the main actor and it is a comedy. Previous utterance: That sounds like a great movie. Any more details? Answer: The screenplay came out in 1990 and has been on the air for quite a while. Toy Story: Tom Hanks as Woody | animated buddy comedy | Toy Story was the first feature length computer animated film | produced by Pixar | toys pretend to be lifeless whenever humans are present | focuses on the relationship between Woody and Gold | fashioned pull string cowboy doll La La Land: Ryan Gosling | Emma Stone as Mia | Hollywood | the city of Los Angeles | Meta critics: 93/100 | 2016 | During a gig at a restaurant Sebastian slips into a passionate jazz | despite warning from the owner | Mia overhears the music as she passes by | for his disobedience Figure 5: Phrases that hold larger spaces of the LTM, when using sticky memories, for two dialogue examples of hard-coded Gaussian attention as input-agnostic self-attention layer for machine translation; Dubois et al. (2020) proposed using Gaussian attention as a location attention mechanism to improve the model generalization to longer sequences. However, these approaches still consider discrete sequences and compute the attention by evaluating the Gaussian density at the token positions. Farinhas et al. (2021) extend continuous attention to multimodal densities, i.e. , mixtures of Gaussians, and applied it to VQA. In this paper, we opt for the simpler case, an unimodal Gaussian, and leave sparse and multimodal continuous attention for future work. Efficient transformers. Several methods have been proposed that reduce the transformer's attention complexity, and can, consequently, model longer contexts. Some of these do so by performing sparse attention, either by selecting pre-defined attention patterns (Child et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020), or by learning these patterns from data (Kitaev et al., 2020; Vyas et al., 2020; Tay et al., 2020a; Roy et al., 2021; Wang et al., 2021). Other works focus on directly reducing the attention complexity by applying the (reversed) kernel trick (Katharopoulos et al., 2020; Choromanski et al., 2021; Peng et al., 2021; Jaegle et al., 2021). Closer to our approach are the transformer-XL and compressive transformer models (Dai et al., 2019; Rae et al., 2019), which extend the vanilla transformer with a bounded memory. Memory-augmented language models. RNNs, LSTMs, and GRUs (Hochreiter et al., 1997; Cho et al., 2014) have the ability of keeping a memory state of the past. However, these require backprop-agation through time which is impractical for long sequences. Because of this, Graves et al. (2014), Weston et al. (2014), Joulin and Mikolov (2015) and Grefenstette et al. (2015) proposed extending RNNs with an external memory, while Chandar et al. (2016) and Rae et al. (2016) proposed efficient procedures to read and write from these memories, using hierarchies and sparsity. Grave et al. (2016) and Merity et al. (2017) proposed the use of cache-based memories which store pairs of hidden states and output tokens from previous steps. The distribution over the words in the memory is then combined with the distribution given by the language model. More recently, Khandelwal et al. (2019) and Yogatama et al. (2021) proposed using nearest neighbors to retrieve words from a key-based memory constructed based on the training data. Similarly, Fan et al. (2021) proposed retrieving sentences from a memory based on the training data and auxiliary information. Khandelwal et al. (2019) proposed interpolating the retrieved words probability distributions with the probability over the vocabulary words when generating a new word, while Yogatama et al. (2021) and Fan et al. (2021) proposed combining the information at the architecture level. These models have the disadvantage of needing to perform a retrieval step when generating 5475 each token / utterance, which can be computationally expensive. These approaches are orthogonal to the  -former's LTM and in future work the two can be combined. In this paper, we proposed the  -former: a transformer extended with an unbounded long-term memory. By using a continuous-space attention framework, its attention complexity is independent of the context's length, which allows the model to attend to arbitrarily long contexts while keeping a fixed computation budget. By updating the memory taking into account past usage, the model keeps sticky memories, enforcing the persistence of relevant information in memory over time. Experiments on a sorting synthetic task show that  former scales up to long sequences, maintaining a high accuracy. Experiments on language modeling and document grounded dialogue generation by fine-tuning a pre-trained language model have shown improvements across several metrics. Transformer models that attend to long contexts, to improve their generation quality, need large amounts of computation and memory to perform self-attention. In this paper, we propose an extension to a transformer model that makes the attention complexity independent of the length of the context being attended to. This can lead to a reduced number of parameters needed to model the same context, which can, consequently, lead to gains in efficiency and reduce energy consumption. On the other hand, the  -former, as well as the other transformer language models, can be used on questionable scenarios, such as the generation of fake news (Zellers et al., 2019), defamatory text (Wallace et al., 2019), or other undesired content. This work was supported by the European Research Council (ERC StG DeepSPIN 758969), by the P2020 project MAIA (contract 045909), by the Fundao para a Cincia e Tecnologia through project PTDC/CCI-INF/4703/2021 (PRELUNA, contract UIDB/50008/2020), by the EU H2020 SELMA project (grant agreement No 957017), and by contract PD/BD/150633/2020 in the scope of the Doctoral Program FCT PD/00140/2013 NET-SyS. We thank Jack Rae, Tom Schaul, the SARDINE team members, and the reviewers for helpful discussion and feedback.
abstain abstain objective abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain objective abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective objective objective objective objective abstain objective method objective objective objective objective other other other other other other other other other other other other other other other other other other other other abstain other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other abstain other other other method other other other other method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other abstain other other other other other other other other other abstain other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other