There is mounting evidence that existing neural network models, in particular the very popular sequence-to-sequence architecture, struggle to systematically generalize to unseen compositions of seen components. We demonstrate that one of the reasons hindering compositional generalization relates to representations being entangled . We propose an extension to sequence-to-sequence models which encourages disentanglement by adaptively re-encoding (at each time step) the source input. Specifically, we condition the source representations on the newly decoded target context which makes it easier for the encoder to exploit specialized information for each prediction rather than capturing it all in a single forward pass. Experimental results on semantic parsing and machine translation empirically show that our proposal delivers more disentangled representations and better generalization. 1 1 Introduction When humans use language, they exhibit compositional generalization; they are able to produce and understand a potentially infinite number of novel linguistic expressions by systematically combining known atomic components (Chomsky, 2014; Montague, 1970). For example, if a person knows the meaning of the utterance A boy ate the cake on the table in a house and the verb like, it is natural for them to understand the utterance A boy likes the cake on the table in a house when they encounter it for the first time (see Table 1). Humans are also adept at recognizing novel combinations of familiar syntactic structure, e.g., they would have no trouble processing the above sentence if the preposition beside the tree were added to it, despite not having previously seen the phrase in a house beside the tree (see Table 1). Training Set A boy ate the cake on the table in a house. *cake( x 4 ); *table( x 7 ); boy(x 1 ) AND eat.agent(x 2 , x 1 ) AND eat.theme(x 2 , x 4 ) AND cake.nmod.on(x 4 , x 7 ) AND table.nmod.in(x 7 , x 10 ) AND house(x 10 ) Test Set (Lexical Generalization) A boy likes the cake on the table in a house. Test Set (Structural Generalization) A boy ate the cake on the table in a house beside the tree. *cake(x 4 ); *table(x 7 ); *tree(x 13 ); boy(x 1 ) AND eat.agent(x 2 , x 1 ) AND eat.theme(x 2 , x 4 ) AND cake.nmod.on(x 4 , x 7 ) AND table.nmod.in(x 7 , x 10 ) AND house(x 10 ) AND house.nmod.beside(x 10 , x 13 ) Table 1: Examples from COGS (Kim and Linzen, 2020) showcasing lexical and structural generalization. There has been a long standing debate whether this systematicity can be captured by connectionist architectures (Fodor and Pylyshyn, 1988; Marcus, 2003; Lake and Baroni, 2018) and recent years have witnessed a resurgence of interest thanks to the tremendous success of neural networks at various natural language understanding and generation tasks (Sutskever et al., 2014; Vaswani et al., 2017; Dong and Lapata, 2016; Jia and Liang, 2016). Mounting evidence, however, suggests that existing models, in particular the very popular sequence-to-sequence architecture, struggle with compositional generalization (Finegan-Dollak et al., 2018; Lake and Baroni, 2018; Keysers et al., 2020; Herzig and Berant, 2021). This failure may be due to spurious 4256 correlations which hinder out-of-distribution generalization (Gururangan et al., 2018; Arjovsky et al., 2019; Sagawa et al., 2020) or limited robustness to perturbations in the input (Cheng et al., 2018). In this paper, we identify an entanglement problem with how different semantic factors (e.g., lexical meaning and semantic relations) are represented in neural sequence models that hurts generalization. In theory, neural networks should represent semantic factors in a disentangled way by virtue of the principle of compositionality (Frege, 1884; Partee, 1995) which implies that semantic properties of syntactic constituents are to a certain extent context invariant and the semantic primitives they express are conditionally independent. Disentangled meaning representations ought to preserve this conditional independence, and neural units modeling a particular semantic factor should be relatively invariant to changes in other factors (Bengio et al., 2013). For example, the relation between table and house in Table 1 and its representation should not be affected by whether there is a PP modifying house. However, in a standard neural encoder (e.g., transformer-based) semantic factors tend to be entangled so that changes in one factor affect the representation of others. We further illustrate this problem in an artificial setting and find that a simple marking strategy enhances the learning of disentangled representations. Motivated by this finding, we propose an extension to sequence-to-sequence (seq2seq) models which allows us to learn disentangled representations for compositional generalization. Specifically, at each time step of the decoding, we adaptively re-encode the source input by conditioning the source representations on the newly decoded target context. We therefore build specialized representations which make it easier for the encoder to exploit relevant-only information for each prediction. Experiments on three benchmarks, namely COGS (Kim and Linzen, 2020), CFQ (Keysers et al., 2020), and CoGnition (Li et al., 2021), empirically verify that our proposal leads to better generalization, outperforming competitive baselines and more specialized techniques. We first shed light on the problem of entangled representations with a toy experiment and then move on to describe our modeling solution. For simplicity, we only focus on relations as the kind of semantic factors a model aims to represent, but the entanglement issue could also exist in representations of other factors, such as lexical meaning. Data Creation Let x = [ e 1 , r 1 , e c , r 2 , e 2 ] denote a sequence of symbols. We want to predict the relation between e 1 and e c , and e c and e 2 , which we denote by y = ( y 1 , y 2 ) , with y 1  L 1 and y 2  L 2 where L 1 are a set of relation labels for y 1 and L 2 are a set of relation labels for y 2 . For simplicity, we set e 1 , e c , and e 2 to the same symbol e (i.e., e 1 , e c , e 2  { e } ) whereas r 1  R 1 and r 2  R 2 denote different relation symbols, and R 1 and R 2 are the corresponding sets of relation candidates. In this toy setting, we will further assume that different relation symbols determine different relation labels (e.g., for the phrases cat in house and cat with house, in and with represent two distinct relations between cat and house). In reality, relations between words could be dependent on broader context or not verbalized at all. We also assume that there is a one-to-one mapping between relation symbols and relation labels (i.e., between L 1 and R 1 and L 2 and R 2 ). We construct a training set by including examples [ e 1 , r 1 , e c , r 2 , e 2 ] where r 1 is the same relation symbol throughout while r 2 can be any relation symbol in R 2 ( r 1  { r train } , r 2  R 2 ). We also include examples [ e 1 , r 1 , e c ] with all relation symbols from R 1 occurring in isolation ( r 1  R 1 ) . This way, the training set covers all primitive relations, but contains only a particular type of relation composition (i.e., { r train }  R 2 ). In contrast, the test set contains all unseen compositions [ e 1 , r 1 , e c , r 2 , e 2 ] (i.e., r 1  R 1 \{ r train } , r 2  R 2 ) which will allow us to evaluate a model's ability to generalize. We set each relation set to include 10 relation symbols ( | R 1 | = | R 2 | = 10). Finally, we simplistically only consider the relations of target word e c with its left and right words e 1 and e 2 . In reality, a model would be expected to capture sentence-level semantics, i.e., a word's relation to all context words in a sentence (including no relation). Modeling For each input symbol, we sample a vector from a Gaussian distribution N ( 0 , 0 . 2 2 I ) and freeze it during training. We then embed each example x into a sequence of vectors [ w 1 , w 2 , ..., w n ] (where n = 3 or n = 5 ) and transform them into contextualized representations [ h 1 , h 2 , .., h n ] using a Transformer encoder 4257 (Vaswani et al., 2017). To predict the relation between two symbols, we concatenate their corresponding representations and feed the resulting vector to an MLP for classification. To study how changes in relation y 1 affect the prediction of y 2 at test time, we explore two training methods. One is joint training where a model learns to predict both y 1 and y 2 (i.e., h 1 and h 3 are concatenated to predict y 1 or h 3 and h 5 are concatenated to predict y 2 ). The other method is separate training where a model is trained to only predict y 2 (i.e., only h 3 and h 5 are concatenated to predict y 2 ) . For separate training, we basically ignore examples [ e 1 , r 1 , e c ] which only include r 1 , as they have no bearing on the prediction of y 2 . Observation With separate training, the model learns to ignore r 1 , the accuracy of predicting y 2 on the test set is 100%, regardless of which value r 1 takes. This indicates that random perturbation of r 1 alone does not lead to generalization failure. It also follows that there is no spurious correlation between r 1 and y 2 . However, when the model is trained to predict both relations (which is what happens in realistic settings since we need to capture all possible relations) r 1 has a huge impact on the prediction of y 2 whose accuracy drops to approximately 55%. Taken together, these results suggest that the model fails to generalize to new relation compositions due to its internal representations being entangled and as a result changes in one relation affect the representation of others. Why is there a wide performance gap between joint and separate training? At test time the model processes the same utterance (no matter whether it is trained jointly or separately), and could in theory be susceptible to both r 1 and r 2 . However, the induced representations show fundamentally different behaviors, and remain invariant to r 1 with separate training. A possible explanation is that modern neural networks trained with SGD have a learning bias towards simple functions (Shah et al., 2020). When r 1 is not predictive of y 2 , relying only on r 2 whilst remaining invariant to r 1 constitutes a simpler function than making use of both r 1 and r 2 . As a result, in separate training the model learns to ignore extraneous information, focusing exclusively on r 2 . On the contrary, in joint training the target of predicting both y 1 and y 2 forces the hidden states (e.g., h 3 ) to capture information about both relations, leading to the entanglement problem discussed above. A Simple Solution Although separate training presents a solution to entanglement, it is unrealistic for real-wold data as it would be extremely inefficient to train separate models for each relation (the number of relations is quadratic with respect to sentence length). Instead, we explore a simple but effective approach where a single model takes as input an utterance enriched with different indicator features for different targets. Specifically, given utterance [ e 1 , r 1 , e c , r 2 , e 2 ] , and assuming we wish to predict relation y 1 , we add indicator feature 1 for symbols e 1 , r 1 , and e c (marking the relation and its immediate context), and 0 for all other symbols. The model then takes as input the utterance and relation indicators, i.e., [1 , 1 , 1 , 0 , 0] for y 1 and [0 , 0 , 1 , 1 , 1] for y 2 , and learns embeddings for indicators during training. It thus learns specialized representations for each prediction rather than shared representations for all predictions. Based on the simplicity bias, the two representations will guide the model towards exclusively relying on r 1 and r 2 , naturally disentangling different relations by encoding them separately. Such a model predicts y 1 with 100% test accuracy and y 2 with 97%. Discussion Fodor and Pylyshyn (1988) have argued that failure to capture systematicity is a major deficiency of neural architectures, contrasting human learners who can readily apply known grammatical rules to arbitrary novel word combinations to individually memorizing an exponential number of sentences. However, our toy experiment shows that neural networks are not just memorizing sentences but implicitly capturing structure. With separate training or joint training enhanced with the marking strategy, the neural model manages to remain robust to interference from r 1 and properly represent r 2 even for unseen examples, i.e., new compositions of r 1 and r 2 . This generalization ability implies that neural models do not need to see all exponential compositions in order to produce plausible representations of them. Instead, with appropriate training and model design, they could uncover and represent the structure underlying systematically related sentences. While the marking strategy offers substantial benefits in learning disentangled relation representations, we typically do not have access to explicit labels indicating which words are helpful for predicting a specific relation. Nevertheless, the idea 4258 of learning representations specialized for different predictions (albeit with shared parameters) is general and could potentially alleviate the entanglement problem for compositional generalization. Let [ x 1 , x 2 , ..., x n ] denote a source sequence. Canonical seq2seq models like the Transformer (Vaswani et al., 2017) first encode it into a sequence of contextualized representations which are then used to decode target symbols [ y 1 , y 2 , ..., y m ] one by one. The same source encodings are used to predict all target symbols, and are therefore expected to capture all semantic factors in the input. However, these could be entangled as demonstrated in our analysis above. To alleviate this issue, we propose to learn specialized source representations for different predictions by adaptively re-encoding the source input at every step of the decoding. Specifically, at the t -th time step, we concatenate the source input with the previously decoded target and obtain the context for the current prediction C t = [ x 1 , x 2 , ..., x n , y 1 , ..., y t  1 , [PH]] where [PH] is a placeholder (e.g., a mask token when using a pretrained encoder). C t is then fed to a standard encoder (e.g., the Transformer encoder) to obtain the contextualized representations H t = [ h t, 1 , h t, 2 , ..., h t,n , h t,n +1 , ..., h t,n + t ] : H t = f Encoder ( C t ) (1) The key difference from the encoder in standard seq2seq models is that at each time step we adaptively re-compute source encodings H t,n = [ h t, 1 , ..., h t,n ] that condition on the newly decoded target [ y 1 , ..., y t  1 ] . This way, target context informs the encoder of predictions of interest at each time step. This simple modification unburdens the model from capturing all source information through a forward pass of encoding. Instead, based on the simplicity bias, the model tends to zero in on information relevant for the current prediction, remaining invariant to irrelevant details, thereby improving disentanglement. One might argue that the decoder in standard seq2seq models could also extract specialized information for each prediction (through the cross attention mechanism). However, it would fail to do so when working with an entangled encoder that produces problematic representations for out-of-distribution examples and breaks down the decoding process. We propose two strategies for exploiting the target-informed encoder. Firstly, we use a multilayer perceptron (MLP) to predict y t based on the encoder's output, i.e., the last hidden states h t,n + t : p ( y t | x, y <t ) = f MLP ( h t,n + t ) (2) Secondly, we incorporate the proposed encoder into the standard encoder-decoder architecture: we take source encodings H t,n and feed them together with the previous target [ y 1 , ..., y t  1 ] to a standard decoder (e.g., Transformer-based) to predict y t : p ( y t | x, y <t ) = f Decoder ( H t,n , y <t ) (3) For complex tasks like machine translation, preserving the encoder-decoder architecture is essential to achieving good performance. We adopt the Transformer architecture to instantiate the encoder and decoder, however, the proposed method is generally applicable to any seq2seq model. We maintain separate position encodings for source and target symbols (e.g., x 1 and y 1 correspond to the same position). To differentiate between source and target content, we also add a source(target) type embedding to all source(target) token embeddings. Compared to the classical Transformer, our proposal increases running time from O ( n 2 + m 2 ) to O ( m ( n 2 + m 2 )) where n is input length and m is output length. Improving the efficiency of our approach is deferred to future work. In this section, we present our experiments for evaluating the proposed D isent angle d seq2seq model which we call DANGLE . We refer to the two variants of DANGLE as DANGLE-ENC and DANGLE-ENCDEC . We first focus on semantic parsing benchmarks which target compositional generalization. Our second suite of experiments reports results on compositional generalization for machine translation. Our semantic parsing experiments focus on two benchmarks. The first one is COGS (Kim and Linzen, 2020) which contains natural language sentences paired with logical forms based on lambda calculus (see the examples in Table 1). In addition to the standard splits of Train/Dev/Test, COGS provides a generalization (Gen) set that covers five types of compositional generalization: interpreting novel combinations of primitives and grammatical roles, verb argument structure alternation, and 4259 sensitivity to verb class, interpreting novel combinations of modified phrases and grammatical roles, generalizing phrase nesting to unseen depths. The former three fall into lexical generalization while the latter two require structural generalization. Interpreting novel combinations of modified phrases and grammatical roles involves generalizing from examples with PP modifiers within object NPs to PP modifiers within subject NPs. The generalization of phrase nesting to unseen depths is concerned with two types of recursive constructions: nested CPs (e.g., [ Mary knows that [ John knows [ that Emma cooks ] CP ] CP ] CP ) and nested PPs (e.g., Ava saw the ball [ in the bottle [ on the table ] PP ] PP ). The training set only contains nestings of depth 02, where depth 0 is a phrase without nesting. The generalization set contains nestings of strictly greater depths (312). The Train set includes 24,155 examples and the Gen set includes 21,000 examples. Our second benchmark is CFQ (Keysers et al., 2020), a large-scale dataset specifically designed to measure compositional generalization. It contains 239,357 compositional Freebase questions paired with SPARQL queries. CFQ was automatically generated from a set of rules in a way that precisely tracks which rules (atoms) and rule combinations (compounds) were used to generate each example. Using this information, the authors generate three splits with maximum compound divergence (MCD) while guaranteeing a small atom divergence between train and test sets. In this dataset atoms refer to entities and relations and compounds to combinations thereof. Large compound divergence indicates the test set contains many examples with unseen syntactic structures. We evaluate our model on all three splits. Each split consists of 95,743/11,968/11,968 train/dev/test examples. On COGS, we trained a baseline TRANSFORMER (Vaswani et al., 2017) with sinusoidal (absolute) and relative position embeddings (Shaw et al., 2018; Huang et al., 2020). We assessed the effect of pretraining on compositional generalization, by also fine-tuning T5BASE (Raffel et al., 2020) on the same dataset. We created disentangled versions of these models adopting an encoder-only architecture (i.e., +D ANGLE-ENC ). The pretrained version of our model used ROBERTA (Liu et al., 2019). 2 2 Note that we use T5BASE instead of ROBERTA as our pretrained baseline on COGS because in initial experiments We also compared with two models specifically designed for compositional generalization on COGS. The first one is TREE-MAML (Conklin et al., 2021), a meta-learning approach whose objective directly optimizes for out-of-distribution generalization. Their best performing model uses tree kernel similarity to construct meta-train and meta-test task pairs. The second approach is LEXLSTM (Akyurek and Andreas, 2021), an LSTM-based seq2seq model whose decoder is augmented with a lexical translation mechanism that generalizes existing copy mechanisms to incorporate learned, decontextualized, token-level translation rules. The lexical translation module is intended to disentangle lexical phenomena from syntactic ones. Furrer et al. (2020) showed that pretrained seq2seq models are key to achieving good performance on CFQ. We compared against their T5-11BMOD model which obtained best results among various pretrained models. This is essentially a T5 model with 11B parameters fine-tuned on CFQ with intermediate representations (i.e., SPARQL queries are simplified to be structurally more aligned to the input for training and then post-processed to obtain the original valid SPARQL at inference time). We also built our model on top of ROBERTA due to the effectiveness of pre-training on this dataset (ROBERTA +D ANGLE-ENC ), again adopting an encoder-only architecture. To tease apart the effect of pretraining and the proposed approach, we also implemented a baseline that makes use of the ROBERTA-BASE model as the encoder and a vanilla Transformer decoder. The Transformer decoder was initialized randomly and trained from scratch. Finally, we compared against HPD (Guo et al., 2020), a hierarchical poset decoding architecture which consists of three components: sketch prediction, primitive prediction, and traversal path prediction. This model is highly optimized for the CFQ dataset and achieves competitive performance. We implemented comparison models and DANGLE with fairseq (Ott et al., 2019); for T5BASE we used HuggingFace Transformers (Wolf et al., 2020). We provide details on model configuration, and various experimental settings in the Appendix. Table 4 shows our results on COGS broken down by type of structural generalization and overall. All models achieve 0 accuracy on generalizing from PP object modifiers to PP subject modifiers. We find this is due to a predicate order bias. In all training examples, agent or theme come before preposition predicates like in, so the models learn this spurious correlation and cannot generalize to cases where the preposition precedes the predicate. Interestingly, a vanilla TRANSFORMER outperforms more complex approaches like TREEMAML and LEXLSTM. We conjecture the large discrepancy is mostly due to our use of Glove embeddings, which comparison systems do not use. Pretraining in general substantially benefits lexical generalization, our TRANSFORMER and T5-B ASE models achieve nearly perfect accuracy on all such cases in COGS. An intuitive explanation is that pretrained embeddings effectively capture common syntactic roles for tokens of the same type (e.g., cat and dog) and facilitate the generalization of the same decoding strategy to all of them. DANGLE-ENC significantly improves generalization performance on CP and PP recursion when combined with our base TRANSFORMER and ROBERTA . additional COGS splits. Table 2 shows how model performance changes with exposure to progressively larger recursion depths. Given recursion depth n , we created a split by moving all examples with depth  n from Gen to Train set. As can be seen, TRANSFORMER +D ANGLE-ENC , especially the variant with relative embeddings, is continuously improving with exposure to additional training examples. In contrast, vanilla TRANSFORMER does not seem to benefit from additional examples, even when relative position encodings are used. We can also explain why adding more recursion in training boosts generalization performance. In the original split, many nouns never occur in examples with recursion depth 2, which could tempt the model to exploit this kind of dataset bias for predictions. In contrast, seeing words in different contexts (e.g., different nesting depth) effectively reduces the possibility of learning these spurious correlations and therefore improves compositional generalization. CFQ results are shown in Table 3. ROBERTA +D ANGLE-ENC substantially boosts the performance of ROBERTA-BASE , and is in fact superior to T5-11BMOD . This result highlights the limitations of pretraining as a solution to compositional generalization underscoring the benefits of our approach. ROBERTA +D ANGLE-ENC is comparable to HPD which is a special-purpose architecture highly optimized for the CFQ dataset. On the contrary, DANGLE is generally applicable to any seq2seq task including machine translation, as we will show in Section 5. As discussed in Section 2, we hypothesize that a neural model's inability to perform compositional generalization partly arises from its internal representations being entangled. To verify this, we visualize the hidden representations for a TRANSFORMER model with and without DANGLE . Specifically, we train both models on the 4th split of COGS (i.e., data with maximum PP 4261 10 5 0 5 10 20 15 10 5 0 5 10 Train Transformer 20 10 0 10 20 25 20 15 10 5 0 5 10 15 Train +Dangle 20 0 20 10 5 0 5 10 Test Transformer 20 10 0 10 20 30 30 20 10 0 10 20 Test +Dangle Figure 1: t-SNE visualization of hidden states corresponding to predicates in, on, and beside on training examples with PP recursion depth 4 and test examples with PP recursion depth 5. Different colors denote different recursion contexts and different shape of markers correspond to different predicates. recursion depth 4) and test on examples with PP recursion depth 5. Then, we extract the hidden states before the softmax layer used to predict the preposition predicates in, beside, and on and use t-SNE (van der Maaten and Hinton, 2008) to visualize them. Ideally, the representations of these prepositions should be invariant to the contexts accompanying them so that their prediction is not influenced by distribution shifts (e.g., contextual changes from PP recursion 4 to PP recursion 5). The visualization is shown in Figure 1. Different colors correspond to different recursion depths while different shape of markers denote different prepositions (e.g., for a training example like NP in NP in NP in NP in NP, the hidden states corresponding to the four in prepositions have the same marker but different colors). In training, TRANSFORMER 's hidden states within the same preposition scatter more widely compared to those of DANGLE , which implies that its internal representations conflate information about a preposi-tion's context with itself. In other words, TRANSFORMER 's hidden states capture more context variations in addition to variations corresponding to the predicate of interest. This in turn causes catastrophic breakdown on the test examples, where TRANSFORMER 's hidden states cannot discriminate context from predicate information at all. This is in stark contrast with DANGLE , where information about predicates is preserved even in the presence of unseen contexts. We further design a metric to quantify entanglement in neural representations drawing inspiration from Kim and Mnih (2018). Their metric assumes the ground-truth factors of a dataset are given, and is applied to images with one factor fixed and all other factors varying randomly; if the representa-COGS CFQ Model IntraV InterV  R IntraV InterV  R TRANSFORMER 0.24 0.64 0.37 0.25 1.13 0.22 +D ANGLE-ENC 0.19 0.73 0.26 0.01 0.52 0.01 TRANSFORMER 0.28 0.44 0.63 0.32 1.06 0.30 +D ANGLE-ENC 0.23 0.54 0.42 0.04 0.48 0.08 Table 5: Entanglement for TRANSFORMER and our approach (+D ANGLE-ENC ) on COGS and CFQ (for which both models employ a ROBERTA encoder). tion is perfectly disentangled, the dimension with the lowest variance should correspond to the fixed factor. Since in our setting we do not have access to ground-truth factors, we assume the variable-length target token sequence is the factor of interest. We also do not need to perform a mapping between neurons and factors, because their correspondence is hard-coded in seq2seq models (e.g., a predicate and the hidden units used to predict it). For each predicate y occurring in different examples e , we extract all corresponding representations { v e,y } , i.e., the last layer of the hidden states used to predict y , and compute the empirical variance Var e ( v ie,y ) for each y ; we compute intra-class variance as the average of all predicates' variances weighted by their respective frequency: V intra = 1 d d (cid:88) i =1 E y Var e ( v ie,y ) (4) where d is the dimension of hidden states and E is the weighted average of their variances. Intuitively, if the representations are perfectly disentangled, they should remain invariant to context changes and intra-class variance should be zero. Inter-class variance, on the contrary, should be relatively large for these hidden states, because they are intended to capture class variations. The ratio of intraand inter-class variance collectively measures entanglement. As shown in Table 5, representations in DANGLE consistently obtain lower intrato inter-class ratios than baseline models on both COGS and CFQ on both training and test sets. We also applied our approach to CoGnition (Li et al., 2021), a recently released realistic compositional generalization dataset targeting machine translation. This benchmark includes 216K English-Chinese sentence pairs; source sentences were taken from the Story Cloze Test and ROCSto-ries Corpora (Mostafazadeh et al., 2016, 2017) and target sentences were constructed by post-editing the output of a machine translation engine. It also contains a synthetic test set to quantify and analyze compositional generalization of neural MT models. This test set includes 10,800 sentence pairs, which were constructed by embedding synthesized novel compounds into training sentence templates. Table 6 shows an example. Each newly constructed compound is combined with 5 different sentence templates, so that every compound can be evaluated under 5 different contexts. We compared our model to a TRANSFORMER translation model following the same setting and con-Model figuration of Li et al. (2021). Again, we experimented with sinusoidal (absolute) and relative position embeddings. We adopted the encoder-decoder architecture variant of our approach (i.e., DANGLEENCDEC ), as the encoder-only architecture performed poorly possibly due to the complexity of the machine translation task. The number of parameters was kept approximately identical to the TRANSFORMER baseline for a fair comparison. All models were implemented using fairseq (Ott et al., 2019). More modeling details are provided in the Appendix. As shown in Table 7, +D ANGLE-ENCDEC improves over the base TRANSFORMER model by 1.2 BLEU points when relative position embeddings are taken into account. In addition to BLUE, Li et al. (2021) evaluate compositional generalization using novel compound translation error rate which is computed over instances and aggregated over contexts. +D ANGLE-ENCDEC variants significantly reduce novel compound translation errors both across instances and on aggregate by as much as 10 absolute accuracy points (see first two column in Table 7). Across metrics, our results show that +D ANGLE-ENCDEC variants handle compositional generalization better than the vanilla. TRANSFORMER model. Two natural questions emerge given the substantial gain achieved by DANGLE on the compositional generalization (CG) test set: (a) Is this gain related to our treatment of the entanglement problem? and (b) How does entanglement manifest itself in machine translation? We attempt to answer these questions with an example. In the CG test set, five new utterances are constructed by embedding the novel compound "be-hind the small doctor on the floor" into five sen-4263 tence templates. In the training set, the phrases behind the [ADJ] [NOUN] and the [ADJ] [NOUN] on the floor appear frequently, but the phrase behind the [ADJ] [NOUN] the [ADJ] [NOUN] is very rare. This poses a serious challenge for the baseline encoder-decoder model, which mistakenly translates the compound phrase into         (the small doctor behind the floor), or        (the small doctor on the floor), or altogether ignores the translation of some content words like     (behind the floor). It seems the baseline model cannot simultaneously represent the relation between behind and the small doctor and the relation between the small doctor and the floor, even though the two are conditionally independent. In contrast, DANGLE generates the correct translation          in all five contexts. We believe this is due to the proposed adaptive encoding mechanism and its ability to decompose the representation problem of an unfamiliar compound phrase into sub-problems of familiar phrases (i.e, behind the small doctor and the small doctor on the floor). The realization that neural sequence models struggle in settings requiring compositional generalization has led to numerous research efforts aiming to understand why this happens and how to prevent it. One line of research tries to improve compositional generalization by adopting a more conventional grammar-based approach (Herzig and Berant, 2021), incorporating a lexicon or lexicon-style alignments into sequence models (Akyurek and Andreas, 2021; Zheng and Lapata, 2021), and augmenting the standard training objective with attention supervision losses (Oren et al., 2020; Yin et al., 2021). Other work resorts to data augmentation strategies as a way of injecting a compositional inductive bias into neural models (Jia and Liang, 2016; Akyrek et al., 2021; Andreas, 2020) and meta-learning to directly optimize for out-of-distribution generalization (Conklin et al., 2021). There are also several approaches which explore the benefits of large-scale pre-trained language models (Oren et al., 2020; Furrer et al., 2020). In this work we identify the learning of representations which are not disentangled as one of the reasons why neural sequence models fail to generalize compositionally. Disentanglement, i.e., the ability to uncover explanatory factors from data, is often cited as a key property of good representations (Bengio et al., 2013). For example, a model trained on 3D objects might learn factors such as object identity, position, scale, lighting, or colour. Several types of variational autoencoders (Kingma and Welling, 2014) have been proposed for the unsupervised learning of disentangled representations in images (Higgins et al., 2017; Kim and Mnih, 2018; Chen et al., 2018). However, some of the underlying assumptions of these models have come under scrutiny recently (Locatello et al., 2019). Disentanglement for linguistic representations remains under-explored, and has mostly focused on separating the style of text from its content (John et al., 2019; Cheng et al., 2020). In the context of sentence-level semantics, disentangled representations should be able to discriminate among lexical meanings and semantic relations between words. We highlight the entanglement problem in neural sequence models when trained with explicit factor supervision which, however, does not cover the entire exponential space of compositions for different factors. Instead of encouraging disentanglement with some regularization (Higgins et al., 2017; Kim and Mnih, 2018), we propose a modification to sequence-to-sequence models which achieves this by re-encoding the source based on newly decoded target context. It may be counterintuitive that we are disentangling by conditioning on more information, but it is feasible thanks to the inherent simplicity bias in neural models. In this paper we proposed an extension to sequence-to-sequence models which allows us to learn disentangled representations for compositional generalization. We have argued that taking into account the target context makes it easier for the encoder to exploit specialized information for improving its predictions. Experiments on semantic parsing and machine translation have shown that our proposal improves compositional generalization without any model, dataset, or task specific modification. Acknowledgments We thank Chunchuan Lyu, Bailin Wang, and the anonymous reviewers for their useful feedback and Yafu Li for his help with our machine translation experiments. We gratefully acknowledge the support of the European Research Council (award number 681760).
abstain objective objective abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain result objective abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other abstain abstain method method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other method other other other other other other abstain abstain abstain objective result result abstain abstain