Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Transformer models showing that the inductive biases given to the model by several design decisions significantly impact compositional generalization. We identified Transformer configurations that generalize compositionally significantly better than previously reported in the literature in many compositional tasks. We achieve state-of-the-art results in a semantic parsing compositional generalization benchmark (COGS), and a string edit operation composition benchmark (PCFG). Although modern neural network architectures reach state-of-the-art performance in many challenging natural language tasks, they seem to exhibit a low amount of compositional generalization, i.e., the ability to learn a set of basic primitives and combine them in more complex ways than those seen during training (Hupkes et al., 2020). For example, suppose a system has learned the meaning of jump and that jump twice means that the action jump has to be repeated two times. Upon learning the meaning of the action jax, it should be able to infer what jax twice means. Compositional generalization is a key aspect of natural language and many other tasks we might want machine learning models to learn. While both humans and classical AI techniques (such as grammars or search-based systems) can handle compositional tasks with relative ease, it seems that modern deep learning techniques do not possess this ability. A key question is thus: Can we build deep learning architectures that can also solve compositional tasks? In this paper we focus on Transformers (Vaswani et al., 2017), which have been shown in the literature to exhibit poor compositional generalization (see Section 2). Through an empirical study, we show that this can be improved. With the goal of creating general models that generalize compositionally in a large range of tasks, we show that several design decisions, such as position encodings, decoder type, weight sharing, model hyper-parameters, and formulation of the target task result in different inductive biases, with significant impact for compositional generalization 1 . We use a collection of twelve datasets designed to measure compositional generalization. In addition to six standard datasets commonly used in the literature (such as SCAN (Lake and Baroni, 2018), PCFG (Hupkes et al., 2020), CFQ (Keysers et al., 2019) and COGS (Kim and Linzen, 2020)), we also use a set of basic algorithmic tasks (such as addition, duplication, or set intersection) that although not directly involving natural language, are useful to obtain insights into what can and cannot be learned with different Transformer models. We also include tasks where we do not see significant improvements, to understand what types of compositional generalization are improved with our proposed modifications, and which are not. The main contributions of this paper are: (1) A study of the Transformer design space, showing which design choices result in compositional learning biases across a variety of tasks. (2) state-of-the-art results in COGS, where we report a classification accuracy of 0.784 using an intermediate representation based on sequence tagging (compared to 0.35 for the best previously reported model (Kim and Linzen, 2020)), and the productivity and systematicity splits of PCFG (Hupkes et al., 2020). The rest of this paper is organized as follows. Section 2 provides some background on compositional generalization and Transformers. In Sec-1 Source code: https://github.com/ google-research/google-research/tree/master/compositional_transformers . tion 3, we present the datasets used in our empirical evaluation, which is presented in Section 4. The paper closes with a discussion on the implications of our results, and directions for future work. Compositional generalization can manifest in different ways. Hupkes et al. (2020) identified five different types, such as systematicity and productivity (extrapolation to longer sequences than those seen during training). Systematicity is the ability of recombining known parts and rules in different ways than seen during training. The example in the introduction of knowing the meaning of jump, jump twice and jax and from those inferring the meaning of jax twice is an example of systematicity. Productivity , on the other hand, is the ability to extrapolate to longer sequences than those seen during training. For example, consider the example of learning how to evaluate mathematical expressions of the form  3 + (4  (5  2)) . An example of productivity would be to extrapolate to expressions with a larger number of parenthesis, or with deeper parenthesis nesting, than seen during training. Hupkes et al. (2020) identify other forms of compositionality, such as substitutivity , localism or overgeneralization , but we will mostly focus on systematicity and productivity in this paper. Compositional generalization is related to the general problem of out-of-distribution generalization . Hence, we can also see it as the problem of how models can discover symmetries in the domain (such as the existence of primitive operations or other regularities) that would generalize better to out-of-distribution samples than shortcuts (Geirhos et al., 2020), which would only work on the same distribution of examples seen during training. Early work focused on showing how different deep learning models do not generalize compositionally (Lika et al., 2018). For example Lika et al. (2018) showed that while models like LSTMs are able to generalize compositionally, it is unlikely that gradient descent converges to a solution that does so (only about 2% out of 50000 training runs achieved a generalization accuracy higher than 80% in a compositional task, while they had almost perfect performance in training). Datasets like SCAN (Lake and Baroni, 2018), PCFG (Hup-kes et al., 2020), Arithmetic language (Veldhoen et al., 2016), or CFQ (Keysers et al., 2019) were proposed to show these effects. Work toward improving compositional generalization includes ideas like Syntactic attention (Russin et al., 2019), increased pretraining (Furrer et al., 2020), data augmentation (Andreas, 2019), intermediate representations (Herzig et al., 2021) or structure annotations (Kim et al., 2021). Specialized architectures that achieve good performance in specific compositional generalization tasks also exist. For example, Liu et al. (2020) propose a model made up of a composer and a solver, achieving perfect performance on SCAN. The most related concurrent work to ours is that of Csords et al. (2021), who also showed gains in compositional generalization via relative attention. Additionally, in their work, they show that a key problem in some tasks is the end of sequence detection problem (when to stop producing output). Finally, they show that generalization accuracy keeps growing even when training accuracy maxes out, questioning early stopping approaches in compositional generalization. We note that training for longer might also improve our results, which we will explore in the future. Models based on Transformers (Vaswani et al., 2017), such as BERT (Devlin et al., 2018), or variants (Yang et al., 2019; Lan et al., 2019; Raffel et al., 2019) yield state-of-the-art results in many NLP tasks such as language modeling (Child et al., 2019; Sukhbaatar et al., 2019; Rae et al., 2019; Kitaev et al., 2020), question answering (Ainslie et al., 2020; Lan et al., 2019; Zaheer et al., 2020; Beltagy et al., 2020), and summarization (Zhang et al., 2019). However, existing studies show that they do not have good compositional generalization. In this paper we will consider the original Transformer architecture and expand upon it. The standard Transformer model consists of two main components (see the center of Figure 2): an encoder and a decoder , each of which consists of a series of layers. Each layer contains an attention sublayer followed by a feed-forward sublayer (the decoder has two attention sublayers for decoder-to-decoder and decoder-to-encoder attention). The input of a Transformer is a sequence of token embeddings, and the output is a sequence of tokens 3592 Input : # # # 3 6 7 [SEP] # # 1 4 9 1 [END] Output: # # 1 8 5 8 [END] Addition: Input : # # 3 6 7 [SEP] # # 1 4 9 1 [END] Output: # # 1 1 2 4 [END] AdditionNegatives: Input : 1 3 3 7 2 [END] Output: 2 7 3 3 1 [END] Reverse: Input : 1 3 5 7 2 [END] Output: 1 3 5 7 2 1 3 5 7 2 [END] Duplication: Input : 1 2 3 [SEP] a b [END] Output: 1 a [SEP] 2 a [SEP] 3 a [SEP] 1 b [SEP] 2 b [SEP] 3 b [END] Cartesian: Input : a4 b1 f6 [SEP] f7 a4 c3 [END] Output: true [END] Intersection: Input : look around right and walk left twice [END] Output: I_TURN_RIGHT I_LOOK I_TURN_RIGHT I_LOOK I_TURN_RIGHT I_LOOK I_TURN_RIGHT I_LOOK I_TURN_LEFT I_WALK I_TURN_LEFT I_WALK [END] SCAN-length / SCAN-add-jump: Input : swap_first_last copy remove_second E18 E15 Q6 , P15 L18 X10 I15 Y14 [END] Output: Q6 E15 E18 [END] PCFG-productivity / PCFG-systematicity Input : A rose was helped by a dog . generated one at a time by predicting based on the output distribution generated by the decoder. To provide a notion of token order a set of position encodings are typically added to the embedding of each input token to indicate sequence order. We will use l to denote the number of en-coder/decoder layers, d for the dimensionality of token embeddings, f for the intermediate dimensionality used by the feed-forward sublayer, and h for the number of attention-heads in the attention sublayers. The original Transformer model used l = 6 , d = 512 , f = 2048 and h = 8 , as their base configuration. In this paper, we use parameters much smaller than that, as we are evaluating the architectural decisions on relatively small datasets. We use a collection of 12 datasets that require different types of compositional generalization. Six of those dataset consist of algorithmic tasks (addition, reversing lists, etc.), and six of them are standard datasets used to evaluate compositional generalization (most involving natural lan-guage). We note that our algorithmic tasks mostly require productivity -style compositional generalization, while other datasets also require systematicity or synonimity (Hupkes et al., 2020). Specifically, we used the following datasets (see Appendix E for details, and Figure 1 for examples): Addition ( Add ): A synthetic addition task, where the input contains the digits of two integers, and the output should be the digits of their sum. The training set contains numbers with up to 8 digits, and the test set contains numbers with 9 or 10 digits. Numbers are padded to reach a length of 12. AdditionNegatives ( AddNeg ): The same as the previous one, but 25% of the numbers are negative (preceded with the symbol). Reversing ( Reverse ): Where the output is expected to be the input sequence in reverse order. Training contains sequences of up to 16 digits, and the test set contains lengths between 17 to 24. Duplication ( Dup ): The input is a sequence of digits and the output should be the same sequence, repeated twice. Training contains sequences up to 16 digits, and test from 17 to 24. Cartesian ( Cart ): The input contains two sequences of symbols, and the output should be their Cartesian product. Training contains sequences of up to 6 symbols (7 or 8 for testing). Intersection ( Inters ): Given two sequences of symbols, the output should be whether they have a non-empty intersection. Training contains sets with size 1 to 16, and testing 17 to 24. SCAN-length ( SCAN-l ): The length split of the SCAN dataset (Lake and Baroni, 2018). SCAN-add-jump ( SCAN-aj ): The add primitive jump split of the SCAN dataset (Lake and Baroni, 2018). PCFG-productivity ( PCFG-p ): The productivity split of the PCFG dataset (Hupkes et al., 2020) PCFG-sytematicity ( PCFG-s : The systematicity split of the PCFG dataset (Hupkes et al., 2020). COGS : The generalization split of the COGS semantic parsing dataset (Kim and Linzen, 2020). CFQ-mcd1 ( CFQ ): The MCD1 split of the CFQ dataset (Keysers et al., 2019). the training and test sets come from the same distribution, and most Transformer models achieve near 100% accuracy (except a few hard tasks like the Cartesian product or set intersection). Hence, splitting train and test data in a way that requires compositional generalization is key (e.g., having examples with larger sequences in the test set than in the training set). We want to make sure models do not just learn shortcuts (Geirhos et al., 2020) that do not generalize to out-of-distribution data. In this section we present an evaluation of the compositional generalization abilities of Transformers with different architectural configurations. Specifically we evaluated: (1) the type of position encodings, (2) the use of copy decoders, (3) model size, (4) weight sharing, and (5) the use of intermediate representations for prediction (see Figure 2). For this systematic experimentation, we used small Transformer models, without pre-training (all models are trained from scratch). Even if previous work has reported benefits of pre-training in some compositional tasks (e.g., in CFQ (Furrer et al., 2020)), we aim at disentangling the effects of each architecture decision in and of itself, in the search for compositional inductive biases. Our results show that, while these decisions do not affect certain types of compositional generalization tasks, we see significant gains in others. We report the average of at least 3 training runs (for algorithmic tasks, we use at least 5 training runs, and 10 for set intersection since they have a higher variance; see Appendix B). We use sequence-level accuracy as the evaluation metric: an output sequence with even just a single wrong token is considered wrong. While the original Transformer model (Vaswani et al., 2017) and BERT (Devlin et al., 2018) used absolute position encodings , later models such as T5 (Raffel et al., 2019) or ETC (Ainslie et al., 2020) use relative position encodings (Shaw et al., 2018). Relative position encodings assign a label to each pair of tokens in the input (typically representing their relative distance in the input, up to a maximum radius). So, there is a label used for tokens attending to a token two positions to the right, etc. One interesting thing about relative position encodings is that they are position invariant , i.e. two tokens that are k positions apart will attend to each other in the same way, regardless of where they are in the sequence, and hence allowing models to capture further symmetries in the domain. We compare the following position encodings: abs : sinusoidal absolute position encodings (as used in the original Transformer) 2 . rel-e : relative position encodings, where the relative position label defines a learnable embedding that is added to the key during the attention process. We used a maximum local attention radius of 16, which means that we have the following relative position labels { l  16 , l  15 , ..., l  1 , l 0 , l 1 , ..., l 15 , l 16 } . Tokens that are further than 16 positions apart get the l  16 or l 16 labels. rel-b : relative positions define a learnable bias that is added to the attention weight of each attention pair. This is the attention mechanism used by T5 (although they use a logarithmic scheme for representing relative positions). While relative positions are straightforward for encoder-to-encoder and decoder-to-decoder attention, it is unclear what the relative positions should be for decoder-to-encoder. Hence, we tested three alternatives ( rel2-e , rel2-b and rel2-eb in our result tables). rel-* methods do not use relative position labels in decoder to encoder attention, while those named rel2-* do (where token y i in the decoder attending to token x j in the encoder will have label l j  i . Table 1 shows sequence-level classification accuracy for small Transformers ( l = 2 , d = 64 , f = 256 , h = 4 ). The right-most column shows the average accuracy across all datasets, and we can see that position encodings play a very significant role in the performance of the models. Going from 0.137 accuracy of the model with absolute position encodings up to 0.354 for a model with relative position encodings using embeddings (but no bias term), as well as relative positions for decoder-to-encoder attention. In general almost any type of relative position encodings help, but using embeddings helps more than using bias terms. Moreover, position encodings play a bigger role in algorithmic tasks. For example, in the Add and AddNeg tasks, models go from 0.005 and 0.042 accuracy to almost perfect accuracy (0.988 and 0.830 for the rel2-e model). Moreover tasks like SCAN or CFQ do not seem to be affected by position encodings, and using relative position encodings with only a bias term hurts in PCFG. Many tasks (such as the duplication or PCFG datasets used in our experiments) require models able to learn things like output whatever is in position k of the input, rather than having to learn hard-coded rules for outputting the right token, depending on the input, a type of symmetry that can be captured with a copy decoder . The copy decoder in our experiments is fairly simple, and works as follows (Figure 2, top-left). It assumes that the input and output vocabularies are the same (we use the union of input and output vocabularies in our experiments). For a given token x i in the output (with final embedding y i ), in addition to the output probability distribution p 1 over the tokens in the vocabulary, the copy decoder produces a second distribution p 2 , which is then mixed with p 1 via a weight w . p 2 is obtained by attending to the output of the last encoder layer (the attention query is calculated using a learnable weight matrix from y i , the embeddings of the last encoder layer are used as the keys , and the values are a one-hot representation of the input tokens). The result is passed through a softmax layer, resulting in p 2 . Table 2 shows sequence-level classification accuracy for models with and without a copy decoder. As can be seen in the last column ( Avg. ), having a copy decoder consistently helps performance, with all models using a copy decoder ( abs-c , rel-eb-c and rel2-eb-c ) outperforming their counterparts without a copy decoder. Moreover, we see that the copy decoder helps the most in PCFG and COGS , while it does not seem to help in some other tasks. Moreover, we would like to point out that there are other ways to set up copy decoders. For example Akyrek et al. (2021) propose defining a lexical 3595 Add AddNeg Reverse Dup Cart Inters SCAN-l SCAN-aj PCFG-p PCFG-s COGS CFQ Avg. translation layer in the copy decoder, which allows models to translate tokens in the input to tokens in the output (which is useful in tasks such as SCAN, which have disjoint vocabularies). In their work, they propose to initialize this layer via a lexicon learning task. Next, we compare the effect of varying both the number of layers ( l ), as well as their size ( d , f h ). Specifically, we tested models with number of layers l equal to 2, 4 and 6, and layers of two sizes: small ( d = 64 , f = 256 , h = 4 ), and large ( d = 128 , f = 512 , h = 8 ). We denote these models small-2 , small-4 , small-6 , large-2 , large-4 , and large-6 . All of the models in this section are variants of rel2-eb-c , our previous best (see Appendix C for parameter counts of our models). Table 3 shows the sequence-level classification accuracy, showing a few interesting facts. First, in most algorithmic tasks, size does not help. Our hypothesis is that the logic required to learn these tasks does not require too many parameters, and large models probably overfit (e.g., like in Duplication ) 3 . Some datasets, however, do benefit from size. For example, most large models outperform their respective small ones in both variants of PCFG. These results are not unexpected, as most compositional generalization datasets contain idealized examples, often generated via some form of 3 Further investigation showed that lowering the learning rate improves performance in the larger models, preventing the phenomenon seen in the Duplication dataset. Systematically exploring this is left for future work. grammar, and have very small vocabularies (see Table 7). Hence, models might not benefit from size as much as on complex natural language tasks. In this section we evaluate the effect of sharing weights across transformer layers. When weight sharing is activated, all learnable weights from all layers in the encoder are shared across layers, and the same is true across the layers of the decoder . Table 4 shows the resulting performance of the models (to be compared with Table 3). Surprisingly, weight sharing significantly boosts compositional generalization accuracy, and almost all models achieve a higher average accuracy across all datasets than their equivalent models in Table 3. In particular, datasets such as AdditionNegatives see a significant boost, with several models achieving higher than 0.9 accuracy (0.982 for large-6s ). PCFG also significantly benefits from weight sharing, with the large-6s model achieving 0.634 and 0.828 in the productivity and systematicity versions, respectively. These are higher than previously reported results in the literature (using the original Transformer, which is a much larger model): 0.50 and 0.72 (Hupkes et al., 2020). Notice, moreover that achieving good results in PCFG (or SCAN) is easy with specialized models. The important achievement is doing so with general purpose models. Our hypothesis is that a model with shared weights across layers might have a more suited inductive bias to learn primitive operations that are applied repeatedly to the input of the transformer (copying, reversing, duplicating, etc.). The key idea of an intermediate representation is to define a different representation of the target output that is easier to generate by the model, but that can be easily mapped to the desired output. Herzig et al. (2021) recently showed very promising results using this technique in several tasks. Defining useful intermediate representations is task-specific and not trivial. Thus we experimented with it in only two datasets: COGS and CFQ (Figure 3). Our intermediate representation for COGs turns the task from seq2seq into a sequence tagging task. We ask the model to produce 5 tags for each input token: a parent , the role of the relation between the token and its parent (if applicable), the category , the noun determiner (for nouns) and the verb name (for verbs). With these tags, the original output can be constructed deterministically. One of the main advantages of this is that the model is naturally pushed to produce outputs with the correct length even for longer inputs (improving productivity ). For the sequence tagging formulation, we used only the encoder part of the Transformer and added five prediction heads, to predict each tag. For role , category , noun determiner and verb name , we simply had a dense layer with a Sigmoid activation function. For the parent tag, we experimented with 3 different head types: Absolute used a dense layer with a Sigmoid activation to predict the absolute index of the parent in the input sequence ( -1 for no parent). Relative predicted the relative offset of the parent token with respect to the current token, or self for no parent. Finally, Attention used the attention weights from a new attention layer with 1 head to predict the parent. Table 5 shows the experimental results comparing a few configurations of this new tagging approach to a few configurations of the seq2seq approach (see Appendix D for all other configu-rations). Examples in the structural generalization tasks are typically longer than in the train-seq2seq tagging Model abs rel2-eb-c abs rel-eb Size small-2 small-6s small-2 small-2s Parent encoding absolute attention Lexical Generalization: Primitives and Grammatical Roles Subject  Object (common noun) 0.309 0.899 0.911 0.969 Subject  Object (proper noun) 0.098 0.429 0.630 0.826 Object  Subject (common noun) 0.790 0.936 0.982 0.978 Object  Subject (proper noun) 0.207 0.951 0.993 0.995 Prim noun  Subject (common noun) 0.240 0.913 0.993 0.988 Prim noun  Subject (proper noun) 0.019 0.772 0.974 0.996 Prim noun  Object (common noun) 0.017 0.902 0.950 0.953 Prim noun  Object (proper noun) 0.000 0.513 0.651 0.700 Prim verb  Infinitival argument 0.000 0.766 0.000 0.001 Lexical Generalization: Verb Argument Structure Alternation Active  Passive 0.604 0.000 0.697 0.948 Passive  Active 0.196 0.001 0.535 0.897 Object-omitted transitive  Transitive 0.275 0.003 0.527 0.926 Unaccusative  Transitive 0.069 0.003 0.528 0.787 Double object dative  PP dative 0.819 0.000 0.590 0.958 PP dative  Double object dative 0.404 0.004 0.771 0.850 Lexical Generalization: Verb Class Agent NP  Unaccusative Subject 0.399 0.951 0.784 1.000 Theme NP  Obj-omitted trans Subj 0.688 0.965 0.791 0.701 Theme NP  Unergative subject 0.694 0.966 0.930 0.771 Structural Generalization: Phrases and Grammatical Roles Obj-mod PP  Subj-mod PP 0.000 0.000 0.000 0.299 Structural Generalization: Deeper Recursion Depth generalization: PP modifiers 0.003 0.000 0.138 0.681 Depth generalization: Sentential comp 0.000 0.000 0.000 0.233 Overall 0.278 0.475 0.637 0.784 Table 5: Sequence-level accuracy in different generalization subsets in COGS for both seq2seq and sequence tagging models. ing set and require productivity . All the models tested in the original COGS paper (Kim and Linzen, 2020) (and all of our seq2seq approaches above) achieved 0 accuracy in this category. The small-6s seq2seq model improves the overall performance from 0.278 to 0.475, but curiously has near 0 performance on Verb Argument Structure Alternation tasks, worse than the base abs model. The intermediate representation based on tagging works much better. The base abs tagging model manages to get non-zero performance on one structural generalization task, which suggests that enforcing the right output length helps. Finally, when predicting the parent directly from attention weights, the structural generalization tasks score 0.2-0.7, compared to our previous near 0 scores (see Appendix D for common types of errors). one model reaching 0.784, higher than any previously reported performance in COGS in the literature, to the best of our knowledge. This suggests that the encoder has the power to parse the input correctly, but maybe the decoder is not capable of generating the correct output sequence from the encoder in the full transformer. One of the difficulties in the CFQ dataset is that models need to learn to perform Cartesian products (e.g., for questions like who directed and acted in M1 and M2?, the model needs to expand to directed M1, directed M2, acted in M1 and acted in M2). However, as shown in our experiments above, this is a very hard task to learn. Hence, we followed the same idea as in Herzig et al. (2021), and defined an intermediate representation that removes the need to learn Cartesian products by allowing triples of the form (entity list) (relation list) (entity list) . Table 6 shows the sequence-level classification accuracy for models on CFQ and on the version with intermediate representations ( CFQ-im ). While the different variations on Transformer models have little affect on the performance, the use of an intermediate representation significantly improves performance, going from around 0.3 accuracy for most Transformer models to over 0.5, and up to 0.555 for the rel-eb model. This is consistent with the results reported by Herzig et al. (2021). An overall trend is that algorithmic tasks seem to be greatly affected by the different architecture design decisions we explored. In all datasets, except for Cartesian product, there is at least one combination in our experiments that achieved high performance (close to 0.8 accuracy or higher). Cartesian products remain an open challenge for future work, where one of the big obstacles is learning to produce much longer outputs than seen during training (output is quadratic with respect to input size). There are some datasets, such as SCAN-aj , where we did not see large improvements in performance. The main obstacle is learning to handle a symbol (jump) having seen it very few times (or even just once) during training (this also happens in some types of generalization in COGS). None of the variations we experimented with were enough to handle this type of compositionality either. In conclusion, we observed: 1. relative position encodings (when both embeddings and biases are used) seem to never be detrimental (they either provided gains, or did not affect). Results indicate this significantly helps in productivity . Moreover, for tasks where positional information is important (such as addition, or reversing), adding positional encodings to decoder2encoder attention provided significant benefits. Finally, as Table 1 shows, for relative position embeddings to be beneficial, using embeddings was necessary; only using relative position biases was not enough. 2. Adding a copy decoder was generally benefi-cial. some tasks (e.g., Reverse ), but these are high variance tasks (see Table 10 in the Appendix), where results are more uncertain. 3. Model size in terms of embedding dimensions, helped generally. Going from 2 to 4 layers provided a slight benefit in general. Our experiments show going to 6 layers hurt performance, but as noted earlier, additional (un-reported preliminary) experiments indicated larger models might need smaller learning rates, with which they also seem to improve performance (systematic exploration of this is future work). 4. Weight sharing seems to benefit in tasks where there are a clear set of primitives that have to be learned (PCFG in particular), or algorithmic tasks, but it seems to hurt in COGs. Hence, weight sharing does not provide general benefits as the previous modifications. 5. Intermediate representations, although dataset-specific, significantly help when they can be defined, as expected. This paper presented an empirical study of the design space of Transformer models, evaluated in a collection of benchmarks for compositional generalization in language and algorithmic tasks. Our results show that, compared to a baseline Transformer, significant gains in compositional generalization can be achieved. Specifically, the baseline Transformer achieved an average sequence-level accuracy of 0.137, while we showed this can increase to up to 0.527 with some design changes. Accuracy levels of up to 0.493 can be achieved without increasing the parameter count of our baseline model (see Appendix C for parameter counts). Moreover, we achieved state-of-the-art results in COGS (at the time of submission), showing 0.784 accuracy on the generalization set, and two PCFG splits (0.634 and 0.828 respectively). This shows that a key factor in training models that generalize compositionally is to provide the right inductive biases. As part of our future work, we want to explore more dimensions, such as pre-training and optimizer parameters, and study the implications of our results in compositional generalization in large models on real world tasks.
abstain objective method result abstain abstain abstain abstain abstain objective result result objective method result objective objective result method abstain other method result other other other other other method other abstain other other other other other other other other other objective other other method other other other other other method other method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain result abstain result abstain method result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method result result result result abstain objective