Current sequence-to-sequence models are trained to minimize cross-entropy and use softmax to compute the locally normalized probabilities over target sequences. While this setup has led to strong results in a variety of tasks, one unsatisfying aspect is its length bias: models give high scores to short, inadequate hypotheses and often make the empty string the argmaxthe so-called cat got your tongue problem. Recently proposed entmax-based sparse sequence-to-sequence models present a possible solution, since they can shrink the search space by assigning zero probability to bad hypotheses, but their ability to handle word-level tasks with transformers has never been tested. In this work, we show that entmax-based models effectively solve the cat got your tongue problem, removing a major source of model error for neural machine translation. In addition, we generalize label smoothing, a critical regularization technique, to the broader family of Fenchel-Young losses, which includes both cross-entropy and the entmax losses. Our resulting label-smoothed entmax loss models set a new state of the art on multilingual grapheme-to-phoneme conversion and deliver improvements and better calibration properties on cross-lingual morphological inflection and machine translation for 7 language pairs. Sequence-to-sequence models ( seq2seq : Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) have become a powerful and flexible tool for a variety of NLP tasks, including machine translation (MT), morphological inflection (MI; Faruqui et al., 2016), and grapheme-to-phoneme conversion (G2P; Yao and Zweig, 2015). These models often perform well, but they have a bias that favors short hypotheses. This bias is problematic: it has been pointed out as the cause (Koehn and Knowles, 2017; Yang et al., 2018; Murray and Chiang, 2018) of the beam search curse , in which increasing the width of beam search actually decreases performance on neural machine translation (NMT). Further illustrating the severity of the problem, Stahlberg and Byrne (2019) showed that the highest-scoring target sequence in NMT is often the empty string, a phenomenon they dubbed the cat got your tongue problem. These results are undesirable because they show that NMT models' performance depends on the search errors induced by a narrow beam. It would be preferable for models to assign higher scores to good translations than to bad ones, rather than to depend on search errors to make up for model errors. The most common way to alleviate this shortcoming is by altering the decoding objective (Wu et al., 2016; He et al., 2016; Yang et al., 2018; Meister et al., 2020a), but this does not address the underlying problem: the model overestimates the probability of implausible hypotheses. Other solutions use alternate training strategies (Murray and Chiang, 2018; Shen et al., 2016), but it would be preferable not to change the training algorithm. In this paper, we propose a solution based on sparse seq2seq models (Peters et al., 2019), which replace the output softmax (Bridle, 1990) with the entmax transformation. Entmax, unlike softmax, can learn locally sparse distributions over the target vocabulary. This allows a sparse model to shrink the search space : that is, it can learn to give inadequate hypotheses zero probability, instead of counting on beam search to prune them. This has already been demonstrated for MI, where the set of possible hypotheses is often small enough to make beam search exact (Peters et al., 2019; Peters and Martins, 2019). We extend this analysis to MT: although exact beam search is not possible for this large vocabulary task, we show that entmax models prune many inadequate hypotheses, effectively solving the cat got your tongue problem. entmax is that it is not compatible with label smoothing (Szegedy et al., 2016), a useful regularization technique that is widely used for transformers (Vaswani et al., 2017). We solve this problem by generalizing label smoothing from the cross-entropy loss to the wider class of Fenchel-Young losses (Blondel et al., 2020), which includes the entmax loss as a particular case. We show that combining label smoothing with entmax loss improves results on both characterand word-level tasks while keeping the model sparse. We note that, although label smoothing improves calibration, it also exacerbates the cat got your tongue problem regardless of loss function. To sum up, we make the following contributions: 1  We show empirically that models trained with entmax loss rarely assign nonzero probability to the empty string, demonstrating that entmax loss is an elegant way to remove a major class of NMT model errors. We generalize label smoothing from the cross-entropy loss to the wider class of Fenchel-Young losses, exhibiting a formulation for label smoothing which, to our knowledge, is novel. We show that Fenchel-Young label smoothing with entmax loss is highly effective on both characterand word-level tasks. Our technique allows us to set a new state of the art on the SIGMORPHON 2020 shared task for multilingual G2P (Gorman et al., 2020). It also delivers improvements for crosslingual MI from SIGMORPHON 2019 (McCarthy et al., 2019) and for MT on IWSLT 2017 German  English (Cettolo et al., 2017), KFTT Japanese  English (Neubig, 2011), WMT 2016 Romanian  English (Bojar et al., 2016), and WMT 2014 English  German (Bojar et al., 2014) compared to smoothed and unsmoothed cross-entropy loss. A seq2seq model learns a probability distribution p  ( y | x ) over sequences y from a target vocabulary V , conditioned on a source sequence x . This 1 Our code is available at https://github.com/ deep-spin/S7 . where V  is the Kleene closure of V . This is an intractable problem; seq2seq models depend on heuristic search strategies, most commonly beam search (Reddy et al., 1977). Most seq2seq models are locally normalized, with probabilities that decompose by the chain rule: p  ( y | x ) = | y | (cid:89) i =1 p  ( y i | x, y <i ) . This factorization implies that the probability of a hypothesis being generated is monotonically nonincreasing in its length, which favors shorter sequences. This phenomenon feeds the beam search curse because short hypotheses 2 are pruned from a narrow beam but survive a wider one. The conditional distribution p  ( y i | x, y <i ) is obtained by first computing a vector of scores (or logits) z = f  ( x, y <i )  R | V | , where f  is parameterized by a neural network, and then applying a transformation  : R | V |  (cid:52) | V | , which maps scores to the probability simplex (cid:52) | V | := { p  R | V | : p  0 , (cid:107) p (cid:107) 1 = 1 } . The usual choice for  is softmax (Bridle, 1990), which returns strictly positive values, ensuring that all sequences  V  have nonzero probability. Coupled with the short sequence bias, this causes significant model error. Sparse seq2seq models. In a sparse model, the output softmax is replaced by a transformation  from the entmax family (Peters et al., 2019). Like softmax, entmax transformations return a vector in the simplex and are differentiable (almost) everywhere. However, unlike softmax, they are capable of producing sparse probability distributions . Concretely, this is done by using the so-called   exponential function (Tsallis, 1988) in place of the exponential, where   0 : exp  ( v ) := (cid:40) [1 + (1   ) v ] 1 / (1   ) + ,  (cid:54) = 1 exp( v ) ,  = 1 . The  -exponential function converges to the regular exponential when   1 . Entmax models assume that p ( y i | x, y <i ) results from an  -entmax 2 We use hypothesis to mean any sequence that ends with the special end-of-sequence token. transformation of the scores z , defined as [  entmax ( z )] y := exp 2   ( z y    ( z )) , (4) where   ( z ) is a constant which ensures normalization. When  = 1 , (4) turns to a regular exponential function and  1 ( z ) = log (cid:80) | V | y (cid:48) =1 exp( z y (cid:48) ) is the log-partition function, recovering softmax. When  = 2 , we recover sparsemax (Martins and Astudillo, 2016). For   { 1 . 5 , 2 } , fast algorithms to compute (4) are available which are almost as fast as evaluating softmax. For other values of  , slower bisection algorithms exist. Entmax transformations are sparse for any  > 1 , with higher values tending to produce sparser outputs. This sparsity allows a model to assign exactly zero probability to implausible hypotheses. For tasks where there is only one correct target sequence, this often allows the model to concentrate all probability mass into a small set of hypotheses, making search exact (Peters and Martins, 2019). This is not possible for open-ended tasks like machine translation, but the model is still locally sparse, assigning zero probability to many hypotheses. These hypotheses will never be selected at any beam width . Fenchel-Young Losses. Inspired by the softmax generalization above, Blondel et al. (2020) provided a tool for constructing a convex loss function. Let  : (cid:52) | V |  R be a strictly convex regularizer which is symmetric, i.e. , ( p ) = ( p ) for any permutation  and any p  (cid:52) | V | . 3 Equipped with  , we can define a regularized prediction function    : R | V |  (cid:52) | V | , with this form:    ( z ) = argmax p (cid:52) | V | z (cid:62) p  ( p ) (5) where z  R | V | is the vector of label scores (logits) and  : (cid:52) | V |  R is a regularizer. Equation 5 recovers both softmax and entmax with particular choices of  : the negative Shannon entropy, ( p ) = (cid:80) y  V p y log p y , recovers the variational form of softmax (Wainwright and Jordan, 2008), while the negative Tsallis entropy (Tsallis, 1988) with parameter  , defined as   ( p ) = (cid:40) 1  (   1) (cid:16)(cid:80) y  V p y  1 (cid:17) , if  (cid:54) = 1 (cid:80) y  V p y log p y , if  = 1 , (6) 3 It is instructive to think of  as a generalized negative entropy: for example, as shown in Blondel et al. (2020, Prop. 4), strict convexity and symmetry imply that  is minimized by the uniform distribution. For a more comprehensive treatment of Fenchel-Young losses, see the cited work. recovers the  -entmax transformation in (4), as shown by Peters et al. (2019). Given the choice of  , the Fenchel-Young loss function L  is defined as L  ( z ; q ) :=   ( z ) + ( q )  z (cid:62) q , (7) where q is a target distribution, most commonly a one-hot vector indicating the gold label, q = e y  = [0 , . . . , 0 , 1 (cid:124)(cid:123)(cid:122)(cid:125) y  -th entry , 0 , . . . , 0] , and   is the convex conjugate of  , defined variationally as:   ( z ) := max p (cid:52) | V | z (cid:62) p  ( p ) . The name stems from the Fenchel-Young inequality, which states that the quantity (7) is nonnegative (Borwein and Lewis, 2010, Prop. 3.3.4). When  is the generalized negative entropy, the loss (7) becomes the Kullback-Leibler divergence between q and softmax ( z ) (KL divergence; Kullback and Leibler, 1951), which equals the cross-entropy when q is a one-hot vector. More generally, if     is the negative Tsallis entropy (6), we obtain the  -entmax loss (Peters et al., 2019). Fenchel-Young losses have nice properties for training neural networks with backpropagation: they are non-negative, convex, and differentiable as long as  is strictly convex (Blondel et al., 2020, Prop. 2). Their gradient is  z L  ( z ; q ) =    ( z )  q , (9) which generalizes the gradient of the cross-entropy loss. Figure 1 illustrates particular cases of Fenchel-Young losses considered in this paper. Label smoothing (Szegedy et al., 2016) has become a popular technique for regularizing the output of a neural network. The intuition behind it is that using the gold target labels from the training set can lead to overconfident models. To overcome this, label smoothing redistributes probability mass from the gold label to the other target labels. When the redistribution is uniform, Pereyra et al. (2017) and Meister et al. (2020b) pointed out that this is equivalent (up to scaling and adding a constant) to adding a second term to the loss that computes the KL divergence DKL ( u (cid:107) p  ) between a uniform distribution u and the model distribution p  . While it might seem appealing to add a similar KL regularizer to a Fenchel-Young loss, this is not possible when p  contains zeroes because the KL divergence term becomes infinite. This makes vanilla label smoothing incompatible with sparse models . Fortunately, there is a more natural generalization of label smoothing to Fenchel-Young losses. For (cid:15)  [0 , 1] , we define the Fenchel-Young label smoothing loss as follows: L  ,(cid:15) ( z , e y  ) := L  ( z , (1  (cid:15) ) e y  + (cid:15) u ) . The intuition is the same as in cross-entropy label smoothing: the target one-hot vector is mixed with a uniform distribution. This simple definition leads to the following result, proved in Appendix A: Proposition 1. The Fenchel-Young label smoothing loss can be written as L  ,(cid:15) ( z , e y  ) = L  ( z , e y  )+ (cid:15) ( z y    z )+ C, (11) where C =  ( e y  ) + ((1  (cid:15) ) e y  + (cid:15) u ) is a constant which does not depend on z , and  z := u (cid:62) z is the average of the logits. Furthermore, up to a constant, we also have L  ,(cid:15) ( z , e y  )  L  ( z , e y  ) + L  ( z , u ) , (12) where  = (cid:15) 1  (cid:15) . The first expression (11) shows that, up to a constant, the smoothed Fenchel-Young loss equals the original loss plus a linear regularizer (cid:15) ( z y    z ) . While this regularizer can be positive or negative, we show in Appendix A that its sum with the original loss L  ( z , e y  ) is always non-negative  intuitively, if the score z y  is below the average, resulting in negative regularization, the unregularized loss will also be larger, and the two terms balance each other. Figure 2 shows the effect of this regularization in the graph of the loss  we see that a correct prediction is linearly penalized with a slope of (cid:15) ; the larger the confidence, the larger the penalty. In particular, when  is the Shannon negentropy, this result shows a simple expression for vanilla label smoothing which, to the best of our knowledge, is novel. The second expression (12) shows that it can also be seen as a form of regularization towards the uniform distribution. When   is the Shannon entropy, the regularizer becomes a KL divergence and we obtain the interpretation of label smoothing for cross-entropy provided by Pereyra et al. (2017) and Meister et al. (2020b). Therefore, the same interpretation holds for the entire Fenchel-Young family if the regularization uses the corresponding Fenchel-Young loss with respect to a uniform. Gradient of Fenchel-Young smoothed loss. From Prop. 1 and Equation 9, we immediately obtain the following expression for the gradient of the smoothed loss:  z L  ,(cid:15) ( z , e y  ) = =  z L  ( z , e y  ) + (cid:15) ( e y   u ) =    ( z )  (1  (cid:15) ) e y   (cid:15) u , (13) that is, the computation of this gradient is straightforward by adding a constant vector to the original gradient of the Fenchel-Young loss; as the latter, it only requires the ability of computing the    transformation, which is efficient in the entmax case as shown by Peters et al. (2019). Note that, unlike the gradient of the original entmax loss, the gradient of its smoothed version is not sparse (in the sense that it will not contain many zeroes); however, since u is the uniform distribution, it will contain many constant terms with value  (cid:15)/ | V | . We trained seq2seq models for three tasks: multilingual G2P, crosslingual MI, and MT. These tasks present very different challenges. In G2P and MI, character-level vocabularies are small and there is usually only one correct target sequence. The relative simplicity of these tasks is offset by the small quantity of training data and the strict evaluation: the model must produce exactly the right sequence. This tests Fenchel-Young label smoothing's ability to learn exactly in a low-resource setting. On the other hand, MT is trained with much larger corpora and evaluated with less strict metrics, but uses subword vocabularies with sizes in the tens of thousands and has to manage more ambiguity because sentences typically have many correct translations. Entmax Loss  : this influences the sparsity of the probability distributions the model returns, with  = 1 recovering cross-entropy and larger  values encouraging sparser distributions. We use   { 1 , 1 . 5 , 2 } for G2P and MI, and   { 1 , 1 . 5 } for MT.  Fenchel-Young Label Smoothing (cid:15) : higher values give more weight to the uniform smoothing distribution, discouraging sparsity. We use (cid:15)  { 0 , 0 . 01 , 0 . 02 , . . . , 0 . 15 } for G2P, (cid:15)  { 0 , 0 . 01 , 0 . 05 , 0 . 1 } for MI, and (cid:15)  { 0 , 0 . 01 , 0 . 1 } for MT. We trained all models with early stopping for a maximum of 30 epochs for MI, 15 epochs for WMT 2014 English  German MT, and 100 epochs otherwise, keeping the best checkpoint according to a task-specific validation metric: Phoneme Error Rate for G2P, average Levenshtein distance for MI, and detokenized BLEU score for MT. At test time, we decoded with a beam width of 5. Our PyTorch code (Paszke et al., 2017) is based on JoeyNMT (Kreutzer et al., 2019) and the entmax implementation from the entmax package. 4 4.1 Multilingual G2P Data. We use the data from SIGMORPHON 2020 Task 1 (Gorman et al., 2020), which includes 3600 training examples in each of 15 languages. We train a single multilingual model (following Peters and Martins, 2020) which must learn to apply spelling rules from several writing systems. Results. Multilingual G2P results are shown in Table 1, along with the best previous result (Yu et al., 2020). We report two error metrics, each of which is computed per-language and averaged:  Word Error Rate (WER) is the percentage of hypotheses which do not exactly match the reference. This harsh metric gives no credit for partial matches. Phoneme Error Rate (PER) is the sum of Levenshtein distances between each hypothesis and the corresponding reference, divided by the total length of the references. These results show that the benefits of sparse losses and label smoothing can be combined. Individually, both label smoothing and sparse loss functions (  > 1 ) consistently improve over unsmoothed cross-entropy (  = 1 ). Together, they produce the best reported result on this dataset. Our approach is very simple, as it requires manipulating only the loss function: there are no changes to the standard seq2seq training or decoding algorithms, no language-specific training or tuning, and no external auxiliary data. In contrast, the previous state of the art (Yu et al., 2020) relies on a complex self-training procedure in which a genetic algorithm is used to learn to ensemble several base models. Data. Our data come from SIGMORPHON 2019 Task 1 (McCarthy et al., 2019), which includes datasets for 100 language pairs. Each training set combines roughly 10,000 examples from a high resource language with 100 examples from a (sim-ulated) low resource language. 5 Development and test sets only cover the low resource language. Training. We reimplemented GATEDATTN (Pe-ters and Martins, 2019), an RNN model with separate encoders for lemma and morphological tags. We copied their hyperparameters, except that we used two layers for all encoders. We concatenated the high and low resource training data. In order to make sure the model paid attention to the low resource training data, we either oversampled it 100 times or used data hallucination (Anastasopoulos and Neubig, 2019) to generate synthetic examples. Hallucination worked well for some languages but not others, so we treated it as a hyperparameter. Results. We compare to CMU-03 6 (Anasta-sopoulos and Neubig, 2019), a two-encoder model with a sophisticated multi-stage training schedule. Despite our models' simpler training technique, 6 We specifically use the official task numbers from McCarthy et al. (2019), which are more complete than those reported in Anastasopoulos and Neubig (2019). they performed nearly as well in terms of accuracy, while recording, to our knowledge, the best Levenshtein distance on this dataset. Having shown the effectiveness of our technique on character-level tasks, we next turn to MT. To our knowledge, entmax loss has never been used for transformer-based MT; Correia et al. (2019) used entmax only for transformer attention. Data. We made use of these language pairs:  IWSLT 2017 German  English ( DE  EN , Cettolo et al., 2017): 200k training examples. KFTT Japanese  English ( JA  EN , Neubig, 2011): 330k training examples. WMT 2016 Romanian  English ( RO  EN , Bojar et al., 2016): 610k training examples. WMT 2014 English  German ( WMT 14, Bojar et al., 2014): 4.4 million training examples. We used joint BPE (Sennrich et al., 2016) for all language pairs, 7 with 25,000 merges for WMT 14 and 32,000 merges for all other pairs. Training. We trained transformers with the base dimension and layer settings (Vaswani et al., 2017). We optimized with Adam (Kingma and Ba, 2015) and used Noam scheduling with 20,000 warmup 7 Although English and Japanese have different writing systems, we still found it beneficial to use joint BPE for JA  EN because many subwords occur in both the English and Japanese training corpora. These include many named entities, which are often written with the native form alongside the transliteration. Results. Table 3 reports our models' performance in terms of untokenized BLEU (Papineni et al., 2002), which we computed with SacreBLEU (Post, 2018). The results show a clear advantage for label smoothing and entmax loss, both separately and together: label-smoothed entmax loss is the best-performing configuration on 3 out of 7 language pairs, unsmoothed entmax loss performs best on another 3 out of 7, and they tie on the remaining one. Although label-smoothed cross-entropy is seen as an essential ingredient for transformer training, entmax loss models beat it even without label smoothing for every pair except EN (cid:1) DE . Model error. Stahlberg and Byrne (2019) showed that the bias in favor of short strings is so strong for softmax NMT models that the argmax sequence is usually the empty string. However, they did not consider the impact of sparsity or label smoothing. 8 We show in Table 4 how often the empty string is more probable than the beam search hypothesis. This is an upper bound for how often the empty string is the argmax because there can also be other hypotheses that are more probable than the empty string. The results show that  and (cid:15) both matter: sparsity substantially reduces the frequency with which the empty string is more probable than the beam search hypothesis, while label smoothing usually increases it. Outcomes vary widely with  = 1 . 5 and (cid:15) = 0 . 1 : WMT 14 and DE  EN models did not seriously suffer from the problem, EN (cid:1) RO did, and the other three language pairs differed from one run to another. The optimal label smoothing value with cross-entropy is invariably (cid:15) = 0 . 1 , which encourages the cat got your tongue problem; on the other hand, entmax 8 They trained with transformer-base settings, implying label smoothing, and did not compare to unsmoothed losses. Other inadequate strings. Even if a model rules out the empty string, it might assign nonzero probability to other short, inadequate strings. We investigated this with a depth-limited search inspired by Stahlberg and Byrne (2019)'s exact decoding technique. Unfortunately, the algorithm's exponential runtime made it unfeasible to perform the search for all language pairs, and in particular we found it too slow for the dense search space of cross entropy models, even after applying various optimizations. 9 Therefore, we show results for EN (cid:1) RO entmax loss models in Table 5. These results show the same trend as on the empty string: short strings are usually pruned for entmax loss models with (cid:15) = 0 or (cid:15) = 0 . 01 , but are likely to have a higher score than the beam-decoded hypothesis with (cid:15) = 0 . 1 . Label smoothing and sparsity. Peters et al. (2019) previously showed that RNN models trained with entmax loss become locally very sparse. Table 6 shows that this is true of transformers as well. Label smoothing encourages greater density, although for the densest language pair ( WMT 14) this only equates to an average support size of roughly 3300 out of a vocabulary of almost 30,000 word types. The relationship between density and overestimating the empty string is inconsistent with (cid:15) = 0 . 1 : WMT 14 and DE  EN models become much more dense but rarely overestimate the empty string (Table 4). The opposite occurs for RO  EN : models with (cid:15) = 0 . 1 become only slightly more dense but are much more prone to model error. This suggests that corpus-specific factors influence both sparsity and how easily bad hypotheses can be pruned. 9 Specifically, when we pushed target sequences to the search's internal stack, we ordered them so that those ending in the end-of-sequence symbol would be popped first. We also discarded any sequence whose probability was lower than that of the beam-decoded hypothesis. Calibration. This is the degree to which a model's confidence about its predictions ( i.e. class probabilities) accurately measure how likely those predictions are to be correct. It has been shown (Muller et al., 2019; Kumar and Sarawagi, 2019) that label smoothing improves the calibration of seq2seq models. We computed the Expected Calibration Error (ECE; Naeini et al., 2015) 10 of our MT models and confirmed their findings. Our results, in Table 7, also show that sparse models are better calibrated than their dense counterparts. This shows that entmax models do not become overconfident even though probability mass is usually concentrated in a small set of possibilities. The good calibration of label smoothing may seem surprising in light of Table 4, which shows that label-smoothed models overestimate the probability of inadequate hypotheses. However, ECE depends only on the relationship between model accuracy and the score of the most likely label. This shows the tradeoff: larger (cid:15) values limit overconfidence but make the tail heavier. Setting  = 1 . 5 with a moderate (cid:15) value seems to be a sensible balance. 10 ECE = (cid:80) Mm =1 | B m | N | acc ( B m )  conf ( B m ) | partitions the model's N force-decoded predictions into M evenly-spaced bins and computes the difference between the accuracy ( acc ( B m ) ) and the average probability of the most likely prediction ( conf ( B m ) ) within that bin. We use M = 10 . Label smoothing. Our work fits into a larger family of techniques that penalize model overconfidence. Pereyra et al. (2017) proposed the confidence penalty, which reverses the direction of the KL divergence in the smoothing expression. Meister et al. (2020b) then introduced a parameterized family of generalized smoothing techniques, different from Fenchel-Young Label Smoothing, which recovers vanilla label smoothing and the confidence penalty as special cases. In a different direction, Wang et al. (2020) improved inference calibration with a graduated label smoothing technique that uses larger smoothing weights for predictions that a baseline model is more confident of. Other works have smoothed over sequences instead of tokens (Norouzi et al., 2016; Elbayad et al., 2018; Lukasik et al., 2020), but this requires approximate techniques for deciding which sequences to smooth. MAP decoding and the empty string. We showed that sparse distributions suffer less from the cat got your tongue problem than their dense counterparts. This makes sense in light of the find-ing that exact MAP decoding works for MI, where probabilities are very peaked even with softmax (Forster and Meister, 2020). For tasks like MT, this is not the case: Eikema and Aziz (2020) pointed out that the argmax receives so little mass that it is almost arbitrary, so seeking it with MAP decoding (which beam search approximates) itself causes many deficiencies of decoding. On the other hand, Meister et al. (2020a) showed that beam search has a helpful bias and introduced regularization penalties for MAP decoding that encode it explicitly. Entmax neither directly addresses the faults of MAP decoding nor compensates for the locality biases of beam search, instead shrinking the gap between beam search and exact decoding. It would be interesting, however, to experiment with these two approaches with entmax in place of softmax. We generalized label smoothing from cross-entropy to the wider class of Fenchel-Young losses. When combined with the entmax loss, we showed meaningful gains on character and word-level tasks, including a new state of the art on multilingual G2P. In addition, we showed that the ability of entmax to shrink the search space significantly alleviates the cat got your tongue problem in machine translation, while also improving model calibration. This work was supported by the European Research Council (ERC StG DeepSPIN 758969), by the P2020 programs MAIA and Unbabel4EU (LISBOA-01-0247-FEDER-045909 and LISBOA-01-0247-FEDER-042671), and by the Fundacao para a Ciencia e Tecnologia through contract UIDB/50008/2020. We thank Wilker Aziz, Vlad Niculae, and the anonymous reviewers, for their helpful discussion and feedback.
abstain abstain abstain objective abstain objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain result result objective objective result abstain other other other other other other other other other other other other other other other other other abstain other other abstain other other other other other other other other other other other method other other other other other other abstain other other method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method other abstain other other other other other abstain other other other other other abstain objective result other other