Unlike literal expressions, idioms' meanings do not directly follow from their parts, posing a challenge for neural machine translation (NMT). NMT models are often unable to translate idioms accurately and over-generate compositional, literal translations. In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention patterns for models with English as source language and one of seven European languages as target language. When Transformer emits a non-literal translation  i.e. identifies the expression as idiomatic  the encoder processes idioms more strongly as single lexical units compared to literal expressions. This manifests in idioms' parts being grouped through attention and in reduced interaction between idioms and their context. In the decoder's cross-attention, figurative inputs result in reduced attention on source-side tokens. These results suggest that Transformer's tendency to process idioms as compositional expressions contributes to literal translations of idioms. An idiom is a group of words of which the figurative meaning differs from the literal reading, such as kick the bucket, which means to die, instead of physically kicking a bucket. An idiom's figurative meaning is established by convention and is typically non-compositional  i.e. the meaning cannot be computed from the meanings of the idiom's parts. Idioms are challenging for the task of neural machine translation (NMT) (Barreiro et al., 2013; Isabelle et al., 2017; Constant et al., 2017; Avramidis et al., 2019). On the one hand, figures of speech are ubiquitous in natural language (Colson, 2019). On the other hand, idioms occur much less frequently than their parts, their meanings need to be memorised due to the non-compositionality, context idiom translatedidiom 1 2 3 4 more attention for literal more attention for figurative </s> translatedcontext cross-attentionself-attention Figure 1: How do attention patterns of figurative PIEs that are paraphrased by the model compare to attention patterns of literal PIEs that are translated word for word? and they require disambiguation before translation. After all, not all potentially idiomatic expressions (PIEs) are figurative  e.g. consider When I kicked the bucket, it fell over. Whether PIEs should receive a figurative or literal translation depends on the context. Yet, little is known about neural mechanisms enabling idiomatic translations and methods for improving them, other than data annotation (Za-ninello and Birch, 2020). Related work studies how idioms are represented by Transformer-based language models (e.g. Garca et al., 2021a,b), but those models are not required to output a discrete representation of the idiom's meaning, which is a complicating factor for NMT models. In this work, we analyse idiom processing for pre-trained NMT Transformer models (Vaswani et al., 2017) for seven European languages by comparing literal and figurative occurrences of PIEs. The comparison can help identify mechanics that underlie neural idiom processing to pave the way for methods that improve idiomatic translations. Large-scale analyses of idiom translations suffer 3608 from a lack of parallel corpora (Fadaee et al., 2018). We, therefore, use a monolingual corpus, heuristically label Transformer's translations, and verify the heuristic works as intended through human evaluation, as described in 3. To understand how idioms are represented in Transformer, we firstly apply interpretability techniques to measure the impact of PIEs on the encoder's self-attention and the cross-attention mechanisms (4), as well as the encoder's hidden representations (5). Afterwards, in 6, we intervene in the models while they process idiomatic expressions to show that one can change non-compositional translations into compositional ones. The results indicate that Transformer typically translates idioms in a too compositional manner, providing a word-for-word translation. Analyses of attention patterns  summarised in Figure 1  and hidden representations point to the encoder as the mechanism grouping components of figurative PIEs. Increased attention within the PIE is accompanied by reduced attention to context. When translating figurative PIEs, the decoder relies less on the encoder's output than for literal PIEs. These patterns are stronger for figurative PIEs that the model paraphrases than for sentences that receive an overly compositional translation and hold across the seven European languages. Considering that a recent trend in NLP is to encourage even more compositional processing in NMT (Raunak et al., 2019; Chaabouni et al., 2021; Li et al., 2021, i.a.), we recommend caution. It may be beneficial to evaluate the effect of compositionality-favouring techniques on non-compositional phenomena like idioms to ensure their effect is not detrimental. This section summarises work discussing human idiom comprehension, interpretability studies for NMT, and literature about figurative language processing in Transformer. Idiom comprehension Historically, idioms were considered non-compositional units (Swinney and Cutler, 1979). Two main views ( literal first and direct access ) existed for how humans interpreted them. The former suggests humans attempt a compositional interpretation before considering the figurative interpretation in case of a contextual discrepancy (Bobrow and Bell, 1973; Grice, 1975, 1989). The latter view suggests one can immediately retrieve the non-compositional meaning (Gibbs Jr et al., 1994). The more recent hybrid view posits that idioms are simultaneously processed as a whole  primed by a superlemma (Kuiper et al., 2007)  and word for word (Caillies and Butcher, 2007). The processing speed and retrieval of the figurative meaning depend on the idiom's semantic properties and the context (Cain et al., 2009; Vulchanova et al., 2019). Examples of semantic properties are the conventionality and decomposability of idioms (Nunberg et al., 1994). We do not expect processes in Transformer to resemble idiom processing in humans. Nonetheless, this work helps us determine our focus of study on the role of the surrounding context and the extent to which idioms' parts are processed as a whole. Translating PIEs that are used figuratively is not always straightforward. Baker et al. (1992) discuss strategies for human translators: (i) Using an idiom from the target language of similar meaning and form, (ii) using an idiom from the target language with a similar meaning and a different form, (iii) copying the idiom to the translation, (iv) paraphrasing the idiom or (v) omitting it. In the absence of idioms with similar meanings across languages, (iv) is the most common strategy. Our main focus is on literal translations ( word-for-word transla-tions), and paraphrases . Interpreting Transformer Analyses of Transformer for NMT studied the encoder's hidden representations and self-attention mechanism (e.g. Raganato and Tiedemann, 2018; Tang et al., 2019b; Voita et al., 2019), the cross-attention (e.g. Tang et al., 2019a) and the decoder (e.g. Yang et al., 2020). The encoder is particularly important for the contextualisation of tokens from the source sentence; it acts as a feature extractor (Tang et al., 2019b). The encoder's bottom three layers better represent low-level syntactic features, whereas the top three layers better capture semantic features (Raganato and Tiedemann, 2018). As a result, one would expect the representations in higher layers to be more representative of idiomaticity. Idioms are a specific kind of ambiguity, and whether a word is ambiguous can accurately be predicted from the encoder's hidden representations, as shown by Tang et al. (2019a) for ambiguous nouns. Transformer's cross-attention is not crucial for disambiguating word senses (Tang et al., 2018), but the encoder's self-attention does reflect ambiguity through more distributed attention for ambiguous nouns (Tang et al., 2019a). Tropes in Transformer Various studies examine the Transformer-based language model BERT's (Devlin et al., 2019) ability to capture tropes like metonyms (Pedinotti and Lenci, 2020), idioms (Kurfal and stling, 2020), and multiple types of figurative language (Shwartz and Dagan, 2019). Kurfal and stling (2020) detect idioms based on the dissimilarity of BERT's representations of a PIE and its context, assuming that contextual discrepancies indicate figurative usage. Pedinotti and Lenci (2020) measure whether BERT detects meaning shift for metonymic expressions but find cloze probabilities more indicative than vector similarities. Shwartz and Dagan (2019) find that BERT is better at detecting figurative meaning shift than at predicting implicit meaning  e.g. predicting that a hot argument does not involve temperature. The most recent work studies properties of hidden representations of noun-noun compounds (NCs) and verb-noun compounds (VCs): Garca et al. (2021b) examine (contextualised) word embeddings, including BERT, to compare figurative and literal NC types . They investigate the similarities between (1) NCs and their synonyms, (2) NCs and their components, (3) in-context and out-of-context representations, and (4) the impact of replacing one component in the NC. Surprisingly, idiomatic NCs are quite similar to their components and are less similar to their synonym compared to literal NCs. Moreover, the context of the NC hardly contributes to how indicative its representation is of idiomaticity, which was also shown by Garca et al. (2021a), who measured the correlation between token -level idiomaticity scores and NCs' similarity inand out-of-context. In search of the idiomatic key of VCs (the part of the input that cues idiomatic usage), Nedumpozhimana and Kelleher (2021) train a probing classifier to distinguish literal usage from figurative usage. They then compare the impact of masking the PIE to masking the context on the classifier's performance and conclude that the idiomatic key mainly lies within the PIE itself, although there is some information coming from the surrounding context. We use Transformer models (Vaswani et al., 2017) with English as the source language and one of seven languages as the target language (Dutch, German, Swedish, Danish, French, Italian, Span-ish). Span-ish). 1 Transformer contains encoder and decoder networks with six self-attention layers each and eight heads per attention mechanism. The models are pre-trained by Tiedemann and Thottingal (2020) with the Marian-MT framework (Junczys-Dowmunt et al., 2018) on a collection of corpora (OPUS) (Tiedemann and Thottingal, 2020). 2 We extract hidden states and attention patterns for sentences with PIEs. The analyses presented are detailed for Dutch, after which we explain how the results for the other languages compare to Dutch. 3 Parallel PIE corpora are rare, exist for a handful of languages only, and are limited in size (Fadaee et al., 2018). Rather than rely on a small parallel corpus, we use the largest corpus of English PIEs to date and annotate the translations heuristically. This section provides corpus statistics and discusses the heuristic annotation method. MAGPIE corpus The MAGPIE corpus presented by Haagsma et al. (2020) contains 1756 English idioms from the Oxford Dictionary of English with 57k occurrences. MAGPIE contains identical PIE matches and morphological and syntactic variants, through the inclusion of common modifications of PIEs, such as passivisation (the beans were spilled) and word insertions (spill all the beans). 4 We use 37k samples annotated as fully figurative or literal , for 1482 idioms that contain nouns, numerals or adjectives that are colours (which we refer to as keywords ). Because idioms show syntactic and morphological variability, we focus mostly on the nouns. Verbs and their translation are harder to identify due to the variability. Moreover, idiom indexes are also typically organised based on the nominal constituents, instead of the verbs (Piirainen, 2013). Only the PIE and its sentential context are presented to the model. We distinguish between PIEs and their context using the corpus's word-level annotations. Heuristic annotation method The MAGPIE sentences are translated by the models with beam search and a beam size of five. The translations are labelled heuristically. In the presence of a literal translation of at least one of the idiom's keywords, 1 Our figures refer to these languages using their ISO 639-1 codes, that are nl , de , sv , da , fr , it and es , respectively. the entire translation is labelled as a word-for-word translation, where the literal translations of keywords are extracted from the model and Google translate. When a literally translated keyword is not present, it is considered a paraphrase . 5 Shao et al. (2018) previously analysed NMT translations of 50 Chinese idioms using a similar method and manually curated lists of literal translations of idioms' words to detect literal translation errors. Dankers et al. (2022) use a similar method for 20 English idioms, to track when a word-for-word translation changes into a paraphrased one during training for an English-Dutch ( En-Nl ) NMT model. Table 1 summarises the distribution of these categories for all languages, for the subsets of figurative and literal examples from MAGPIE. Generally, paraphrased translations of figurative PIEs are more appropriate than word-for-word translations, whereas literal PIEs can be translated word for word (Baker et al., 1992). The vast majority of literal PIEs indeed result in word-for-word translations. The subset of figurative samples results in more paraphrases, but  76% is still a word-for-word translation, dependent on the language. Although the statistics are similar across languages, there are differences in which examples are paraphrased. Figure 2 illustrates the agreement 5 The annotation does not evaluate whether paraphrases are correct, which requires expert idiom knowledge in both languages. A paraphrase being provided is a first step to adequately translating idioms and, at present, the only way to detect how the model approaches the task for large datasets. by computing the F 1 -score when using the predictions for figurative instances of one language as the target, and comparing them to predictions from another language. The agreement positively correlates with genetic similarity as computed using the Uriel database (Littell et al., 2017). To assess the quality of the heuristic method, one (near) native speaker per target language annotated 350 samples, where they were instructed to focus on one PIE keyword in the English sentence. Annotators were asked whether (1) the English word was present in the translation (initially referred to as copy), (2) whether there was a literal translation for the word, or (3) whether neither of those options were suited, referred to as the paraphrase. 6 Due to the presence of cognates in the copy category, that category was merged with the word for word category after the annotation. Table 2 summarises the accuracies obtained. Of particular interest are samples that are figurative and paraphrased, since they represent the translations that are treated non-compositionally by the model, as well as instances that are literal and translated word for word, since they represent the compositional translations for non-idiomatic PIE occurrences. These categories have annotation accuracies of  75% and  89% , respectively. During preliminary analyses, an annotation study was conducted for Dutch by annotators from the crowd-sourcing platform Prolific. The annotators and the heuristic method agreed in 83% of the annotated examples, and for 77% of the samples an average of 4 annotators agreed on the label unanimously (see Appendix A for more details). Sentences containing idioms typically yield lower BLEU scores (Fadaee et al., 2018). MAGPIE is a monolingual corpus and does not allow us to compute BLEU scores, but we refer the reader to Appendix G for an exploratory investigation for MAGPIE's idioms using the En-Nl training corpus. We now turn to comparing how literal and figurative PIEs are processed by Transformer. Whether a PIE is figurative depends on the context  e.g. compare in culinary school, I felt at sea  to the sailors were at sea . Within Transformer, contextualisation of input tokens is achieved through the attention mechanisms, which is why they are expected to combine the representations of the idioms' tokens and embed the idiom in its context. This section discusses the impact of PIEs on the encoder's self-attention and the encoder-decoder cross-attention. To assert that the conclusions drawn in this section are not simply explained by shallow statistics of the data used, we recompute the results in Appendix C for (1) a data subset excluding variations of PIEs' standard surface forms, (2) a data subset that includes PIEs that appear in both figurative and literal contexts, (3) a data subset that controls for the number of tokens within a PIE. Qualitatively, these results lead to the same findings. Attention within the PIE For the En-Nl Transformer, Figure 3a visualises the distribution of attention weights in the encoder's self-attention mechanism for incoming weights to one noun contained in the PIE from the remaining PIE tokens. Throughout the figures in the paper, we refer to the subset of sentences that have a figurative PIE and a 1 2 3 4 5 6 layer 0.0 0.2 0.4 0.6 0.8 1.0 a tt e n t i o n paraphrased translation as  fig-par '. The subset of sentences with a literal PIE and a word-for-word translation are indicated by  lit-wfw '. We compare those two subsets, as well as all instances of figurative PIEs ( fig ') to all instances of literal PIEs ( lit ') using the labels from the MAGPIE dataset. Overall, there is increased attention in figurative occurrences of PIEs compared to literal instances. This difference is amplified for the subset of figurative PIEs yielding paraphrased translations. This pattern is consistent for all languages, as is displayed in Figure 3d that presents the difference between the mean attention weights of the figurative, paraphrased instances, and the mean weights of the literal instances translated word for word. 7 In other words, figurative PIEs are grouped more strongly than their literal counterparts. Attention between PIEs and context To examine the interaction between a PIE and its context, we obtain the attention weights from tokens within the PIE to nouns in the surrounding context of size 10 (Figure 3b). 8 Similarly, the attention from the surrounding context to PIE nouns is measured (Fig-ure 3c). There is reduced attention from PIEs to context for figurative instances, which mirrors the effect observed in Figure 3a: increased attention 7 Appendix D details results per language per layer. 8 Throughout the paper, a context size of 10 to the left and 10 to the right or smaller is used, as sentence length permits. within the PIE is accompanied by reduced attention to the context. This pattern is consistent across languages (Figure 3d). From the context to the PIE, the average weight is slightly higher for literal PIEs, but the effect size is small, indicating only a minor impact of figurativeness on the context's attention weights. This will be further investigated in 5. Cross-attention To analyse the encoder-decoder interaction, we decode translations with beam size five, and extract the cross-attention weights for those translations. Afterwards, alignments are computed for the models' predictions by, together with 1M sentences from the OPUS corpus per target language, aligning them using the eflomal toolkit (stling et al., 2016). The alignment is used to measure attention from a token aligned to a PIE's noun to that noun on the source side. 9 Figure 4a presents the attention distribution for the weights that go from the noun's translation to that PIE noun on the source side, for the En-Nl model. There is a stark difference between figurative and literal PIEs, through reduced attention on the source-side noun for figurative PIEs. This difference is particularly strong for the figurative sentences that are paraphrased during the translation: when paraphrasing the model appears to rely less on the source-side noun than when translating word for word. Where does the attention flow, instead? To some extent, to the remaining PIE tokens (Figure 4b). A more pronounced pattern of increased attention on the </s> token is shown in Figure 4c. Similar behaviour has been observed by Clark et al. (2019) for BERT's [SEP] token, who suggest that this indicates a no-operation . In Transformer's cross-attention mechanism, this would mean that the decoder collects little information from the source side. Figure 4d compares the mean attention weights of the seven languages for the figurative inputs that are paraphrased to the literal samples that are translated word for word, confirming that these patterns are not specific to En-Nl translation. Collectively, the results provide the observations depicted in Figure 1. When paraphrasing a figu-9 Automated alignments may be less accurate for paraphrases, and, therefore, we inspect the fig-par alignments: for all languages  34% of those sentences has no aligned word for the PIE noun. Those sentences are excluded. We manually inspect the most frequently aligned words for Dutch, that cover 48% of the fig-par subcategory in Ap. B, and are all accurate. rative PIE, the model groups idioms' parts more strongly than it would otherwise  i.e. it captures the PIE more as one unit. A lack of grouping all figurative PIEs could be a cause of too compositional translations. Increased attention within the PIE is accompanied by reduced interaction with context, indicating that the PIE is translated in a stand-alone manner, contrary to what is expected, namely that contextualisation can resolve the figurative versus literal ambiguity. There is less cross-attention on the source-side PIE and more attention on the </s> token when the model emits the translation of figurative (paraphrased) PIEs. This suggests that even though the encoder cues figurative usage, the decoder retrieves a PIE's paraphrase and generates its translation more as a language model would. Within Transformer, the encoder's upper layers have previously been found to encode semantic information (e.g. Raganato and Tiedemann, 2018). PIEs' hidden states are expected to transform over layers due to contextualisation, and become increasingly more indicative of figurativeness. This section focuses on the impact of PIEs on the hidden states of Transformer's encoder. We firstly discuss how much these hidden states change between layers. Secondly, we measure the influence of a token by masking it out in the attention and analysing the degree of change in the hidden representations of its neighbouring tokens. This analysis is performed to consolidate findings from 4, since the extent to which attention can explain model behaviour is a topic of debate (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). To compare representations from different layers, we apply canonical correlation analysis (CCA) (Hotelling, 1936), using an implementation from Raghu et al. (2017). Assume matrices A  R d A  N and B  R d B  N , that are representations for N data points, drawn from two different sources with dimensionalities d A and d B  e.g. different layers of one network. CCA linearly transforms these subspaces A (cid:48) = W A , B (cid:48) = V B such as to maximise the correlations {  1 , . . . ,  min ( d A ,d B ) } of the transformed subspaces. We perform CCA using > 60 k random token vectors for a previously unused subset of the MAGPIE corpus  the subset of sentences that did not contain nouns in the PIEs  3613 1-2 2-3 3-4 4-5 5-6 layers 0.75 0.80 0.85 CCA s i m il a r i t y to compute the CCA projection matrices W and V . W and V are then used to project new data points before measuring the data points' correlation. The CCA similarity reported in the graphs is the average correlation of projected data points. We do not perform CCA separately per data subset due to the small subset sizes and the impact of vocabulary sizes on CCA correlations for small datasets (see Appendix E). 10 We compute the CCA similarity for hidden states from adjacent layers for PIE and non-PIE nouns. Figurative PIEs in layer l are typically less similar to their representation in layer l  1 compared to literal instances (shown in Figures 5b and 5c). The results for non-PIE nouns (Figure 5a for the En-Nl Transformer) do not differ across data subsets, suggesting that changes observed for figurative PIEs are indeed due to figurativeness. We now compute similarities of representations for the model in two setups: with and without one token masked in the attention mechanism, as suggested by Voita et al. (2019). Masking a token means that other tokens are forbidden to attend to the chosen one. This can reveal whether the attention patterns discussed in 4 are indicative of the 10 Extensions of CCA have been proposed that limit the number of CCA directions over which the correlation is computed, to only include directions that explain a large portion of the variance (Raghu et al., 2017; Morcos et al., 2018). We do not remove directions such as to avoid removing smaller variance components that could still cue figurativeness (the focus of our work). influence tokens have on each other's hidden representations. The first representation is the hidden representation from layer l for a token encoded as usual. The second one is the hidden representation of layer l when applying the first l  1 layers as usual and masking one token in the l th layer. CCA is again performed on separate data, where a non-PIE noun is masked, to provide the projection matrices applied before computing similarities in the remainder of this subsection. Masking a PIE token To estimate the influence of PIE nouns, we first compute the CCA similarity between two representations of tokens from the PIE's context while masking one PIE noun in the attention for one of those representations. Similarly, we measure the influence on other tokens within the PIE when masking one PIE noun. Within the PIE, the impact is the largest for figurative instances (see Figure 6a for En-Nl and 6e for averages over layers for all languages). This is in line with the attention pattern observed. However, whether the impact is the largest on context tokens from figurative or literal instances is dependent on the layer 3614 0.6 0.8 1.0 m a c r o F 1 nl de sv da e 1 2 3 4 5 6 layers 0.6 0.8 1.0 m a c r o F 1 fr e 1 2 3 4 5 6 layers es e 1 2 3 4 5 6 layers it fig-par/lit-wfw fig/lit Figure 7: Macro F 1 -score for probes predicting PIEs' labels. (Figure 6b), suggesting that the slight difference in attention from the context to the PIE observed in 4 need not represent a difference in impact between figurative and literal PIEs. Masking a context token Lastly, we measure the influence of masking a noun in the context of the PIE on PIE tokens and non-PIE tokens. Within the PIE, as shown in Figures 6c and 6e, figurative instances are less affected by the masked context noun compared to literal occurrences of PIEs. Again, this mirrors the patterns observed for attention where there was less attention on the context for figurative PIEs. When masking a non-PIE noun and measuring the impact on non-PIE tokens, one would hardly expect any differences between data subsets, as is confirmed in Figures 6d and 6e. In summary, these analyses confirm most of the trends noted for attention patterns. Intercepting in the attention through masking indicated that for PIE tokens, there is less interaction with the context. However, this does not necessarily mean that the context interacts less with figurative PIEs compared to literal PIEs, even if there was a slight difference in attention (see 4). The CCA analyses furthermore showed that figurative PIEs are distinct from typical tokens in how they change over layers. The previous analyses compared the hidden states for figurative and literal PIEs, but do not use these labels, otherwise. We now train logistic regression probing classifiers (Conneau et al., 2018) to predict the label from hidden representations. The probes' inputs are the hidden states of PIE tokens, and the F 1 -scores are averaged over five folds. All samples from one PIE are in the same fold, such that the classifier is evaluated on PIEs that were absent from its training data. The results (Figure 7) indicate figurativeness can be predicted from these encodings, with performance increasing until the top layer for all languages. F 1 -scores for the embeddings already exceed a random baseline, indicating some idioms are recognisable independent of context. Finally, we use probing classifiers to change models' PIE translations through amnesic probing (Elazar et al., 2021): removing features from hidden states with iterative null-space projection (INLP) (Ravfogel et al., 2020) and measuring the influence of these interventions. INLP trains k classifiers to predict a property from vectors. After training probe i , parametrised by W i , the vectors are projected onto the nullspace of W i . The projection matrix of the intersection of all k null spaces can then remove features found by these classifiers. Using INLP, we train 50 classifiers to distinguish figurative PIEs that will be paraphrased from those to be translated word for word. Afterwards, we run the previously paraphrased PIE occurrences through the model while removing information from the PIE's hidden states using INLP  i.e. information that could be captured by linear classifiers, which need not be the only features relevant to idiomatic translations. Per idiom, we record the percentage of translations that are no longer paraphrased. We report the scores for idioms from four folds and BLEU scores comparing translations that changed label before and after INLP. A fifth fold is used for parameter estimation (Appendix F). Table 3 presents the results. When intervening in the hidden states for all layers l  { 0 , 1 , 2 , 3 , 4 } , the average success rate per PIE ranges from 27% (for Swedish) to 40% (for Spanish). The interventions yield reduced attention within the PIE and increased interaction with the context (see Table 3b for Dutch). Table 3 also provides results for a baseline probe predicting whether the half-harmonic mean of the zipf-frequency of PIE tokens is below or above average. This probe is successful too, 3615 Dutch German Swedish Danish French Italian Spanish Then, brisk again, ' I 'll bear it in mind. ' Entonces, rpido de nuevo, ' Lo tendr en cuenta. ' Entonces, anmate de nuevo, 'Lo tendr en mente'. emphasising how brittle idiomatic translations are: when removing information from the hidden states, the model reverts to compositional translations. Figure 8 provides example translations before and after the application of INLP, while indicating how the attention on the underlined noun changes. Generally, the attention on that noun reduces for tokens other than itself. In summary, when applying INLP to hidden states, the attention patterns resemble patterns for literal tokens more, confirming a causal connection between the model paraphrasing figurative PIEs and the attention. However, amnesic probing cannot change the paraphrases for all idioms; thus, figurativeness is not merely linearly encoded in the hidden states. The probing accuracies differed across layers and suggested figurativeness is more easily detectable in higher layers, which is in line with the changes across layers observed in 5. Idioms are challenging for NMT models that often generate overly compositional idiom translations. To understand why this occurs, we analysed idiom processing in Transformer, using an English idiom corpus and heuristically labelled translations in seven target languages. We compared hidden states and attention patterns for figurative and literal PIEs. In the encoder, figurative PIEs are grouped more strongly as one lexical unit than literal instances and interact less with their context. The effect is stronger for paraphrased translations, suggesting that capturing idioms as single units and translating them in a stand-alone manner aids idiom processing. This finding agrees with results from Zaninello and Birch (2020), who ascertain that encoding an idiom as one word improves translations. It also agrees with the INLP application causing more compositional translations whilst changing the attention. By relying less on the encoder's output, the decoder determines the meaning of figurative PIEs more independently than for literal ones. To improve idiomatic translations, future work could use these insights to make architectural changes to improve the grouping of idioms as single units by training specific attention heads to capture multiword expressions or by penalising overly compositional translations in the training objective. Although we learnt about mechanics involved in idiomatic translations, the vast majority of translations was still word for word, indicating that non-compositional processing does not emerge well (enough) in Transformer. Paradoxically, a recent trend is to encourage more compositional processing in NMT (Chaabouni et al., 2021; Li et al., 2021; Raunak et al., 2019, i.a.). We recommend caution since this inductive bias may harm idiom translations. It may be beneficial to evaluate the effect of compositionality-favouring techniques on non-compositional phenomena to ensure their effect is not detrimental. We are grateful to Rico Sennrich for providing feedback on an earlier version of the paper. Many thanks to Agostina Calabrese, Matthias Lindemann, Gautier Dagan, Irene Winther, Ronald Cardenas, Helena Fabricius-Vieira and Emelie van de Vreken for data annotation and assistance with queries about their native languages. VD is supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1) and the University of Edinburgh, School of Informatics and School of Philosophy, Psychology & Language Sciences. IT acknowledges the support of the European Research Council (ERC StG BroadSem 678254) and the 3616 Dutch National Science Foundation (NWO Vidi 639.022.518). Raymond W Gibbs Jr, Raymond W Gibbs, and Jr Gibbs. 1994. The poetics of mind: Figurative thought, language, and understanding . Cambridge University Press. H. Paul Grice. 1975. Logic and conversation. Syntax and Semantics , 3:4158. H. Paul Grice. 1989. Studies in the Way of Words . Harvard University Press.
abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain result method result abstain abstain abstain abstain abstain method abstain other other other other other other other other abstain abstain other other other other other other other other other abstain other other other other other other other other other other other other other other other other method abstain abstain abstain method method abstain method abstain abstain abstain method method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain