Classifiers in natural language processing (NLP) often have a large number of output classes. For example, neural language models (LMs) and machine translation (MT) models both predict tokens from a vocabulary of thousands. The Softmax output layer of these models typically receives as input a dense feature representation, which has much lower dimensionality than the output. In theory, the result is some words may be impossible to be predicted via argmax, irrespective of input features, and empirically, there is evidence this happens in small language models (Demeter et al., 2020). In this paper we ask whether it can happen in practical large language models and translation models. To do so, we develop algorithms to detect such unargmaxable tokens in public models. We find that 13 out of 150 models do indeed have such tokens; however, they are very infrequent and unlikely to impact model quality. We release our algorithms and code so that others can test their models. 1 1 Introduction Probabilistic multiclass classifiers with a large number of output classes are commonplace in NLP (Chen et al., 2016). For example, the vocabulary size of contemporary LMs and MT models varies from tens to hundreds of thousands (Liu et al., 2020). Recent advances in modelling such large vocabularies have mostly been made by improving neural network feature encoders (Devlin et al., 2019; Conneau et al., 2020). But irrespective of a feature encoder's expressivity (Yun et al., 2020; Raghu et al., 2017), a classifier that linearly maps lower dimensional features to higher dimensional outputs has reduced expressivity (Yang et al., 2018), with consequences that are not well understood. classifiers that have more output classes | C | than features d . For example, MT models often have subword vocabularies of size | C |  30000 , but have d  1024 . The expressivity penalty for such low-rank classifiers is that some output distributions cannot be represented. Demeter et al. (2020) identified this weakness in Softmax LMs, showing that, in theory, some tokens can never be assigned the highest probability for any input, and therefore can never be produced as argmax predictions. 2 We call such tokens unargmaxable (see Figure 1). While Demeter et al. (2020) proposed an algorithm to detect unargmaxable tokens and provided evidence of their existence in small LMs, their proposed algorithm provided no guarantees and they were unable to test large LMs. In this paper we ask: Do unargmaxable tokens exist in large models used in practice? To answer this question, we develop algorithms to identify such tokens unambiguously. We tested 7 LMs and 143 MT models. Out of those, only 13 of the MT models exhibit unargmaxable tokens, and even for those cases the 2 This problem was also studied by Cover (1967) and has an interesting history of independent discovery (Smith, 2014). tokens are all noisy and infrequent. We conclude that although the expressivity constraints of low-rank Softmax may have important ramifications, most practitioners do not need to worry about tokens that are unargmaxable. We provide new tools for them to confirm this on their own models. Our contributions are the following:  We explain how unargmaxable tokens can arise as a consequence of a rank constrained Softmax layer (Softmax Bottleneck). We extend the work of Demeter et al. (2020) with verification algorithms that include the Softmax bias term and provide an exact answer rather than an approximate one. We verify a large number of commonly used publicly available language and translation models for unargmaxable tokens. We release our algorithm so that others can inspect their models. 1 2 Background 2.1 Low-Rank Softmax (Softmax Bottleneck) Neural network layers with higher dimensional outputs than inputs impose low-rank constraints. 3 Such constraints commonly exist as bottlenecks in neural network hidden layers, e.g. autoen-coders (Hinton and Zemel, 1994) and projection heads in multi-head transformers (Bhojanapalli et al., 2020) among others. While bottlenecks make a model less expressive by restricting the functions it can represent, they are desirable both computationally (Papadimitriou and Jain, 2021), since they require less memory and computation than full-rank layers, and as a form of inductive bias, since data is assumed to approximately lie in a low dimensional manifold (McInnes et al., 2018). In contrast, herein we focus on the undesirable properties of a Softmax output layer with a low-rank parametrisation, also known as a Softmax Bottleneck (Yang et al., 2018). The crucial difference is that a Softmax Bottleneck is usually not followed by a non-linear transformation, and as such the rank constraint limits expressivity in a very rigid way by restricting outputs to a subspace. 4 3 A layer can also be low rank if weight vectors are collinear, but we do not consider this case here. This constraint was shown to hurt LM perplexity (Yang et al., 2018) and non-linear augmentations have been proposed as improvements (Yang et al., 2018; Kanai et al., 2018; Ganea et al., 2019). To the contrary, Sainath et al. (2013) used a low-rank factorisation of the softmax layer to reduce the number of parameters in their speech recognition system by 30 50% with no increase in word-error-rate, evidencing that the loss in expressivity does not always impact aggregate metrics. The consequences of the loss in expressivity due to the Softmax Bottleneck vary depending on our perspective. When considering the flexibility of the probability distribution that can be learned, Ganea et al. (2019, Theorem 2) showed that the minimum cross entropy loss achievable decreases as we increase the rank of the Softmax layer weights. In this work we focus on the loss of expressivity from an argmax perspective. To this end, we discretise the output space of Softmax and quantify the loss in expressivity in terms of unrealisable class rankings. From this interpretable perspective we will see that due to the Softmax Bottleneck some rankings are not realisable and unargmaxable classes can arise as a consequence. Demeter et al. (2020) showed that a class is unargmaxable if its Softmax weight vector is interior to the convex hull of the remaining class weight vectors. They did so by proving that the interior class probability is bounded above by the probability of at least one class on the convex hull (see Figure 2 and Cover, 1967, Figure 1). However, in their analysis they did not address Softmax layers that include a bias term. We address this limitation in Section 3, thus enabling us to search 6739 for unargmaxable classes in any released model. To detect whether unargmaxable tokens arise in LMs without a bias term, the authors introduce an approximate algorithm that asserts whether a weight vector is internal to the convex hull. It is approximate since their method had a precision approaching 100% but 68% recall when compared to an exact algorithm (Qhull, Barber et al., 1996) on the first 10 dimensions of a Softmax LM. In Section 3.3 we introduce an exact algorithm to detect unargmaxable tokens with certainty. The authors use their approximate algorithm to show that AWD-LSTM LMs (Merity et al., 2018) steal probability from candidate interior words when contrasted to the probabilities assigned by a smoothed n-gram LM. However, they find that as they increase the dimensionality d of the Softmax weights to 200 , the effect of stolen probability begins to dissipate. This raises the question of whether stolen probability is of importance for neural models used in practice which also have larger Softmax weight dimensionality. Herein we specifically search for unargmaxable tokens in MT and LM models with larger d  [256 , 512 , 1024] . We use the term unargmaxable rather than stolen probability to highlight that we are focussing on whether unargmaxable tokens exist and not whether the probability distibution learned by low-rank Softmax is less flexible. We extend our analysis to MT models since they have more practical use cases than (generative) LMs: if unargmaxable tokens exists in a MT model, then the affected tokens can never be produced when using greedy decoding. In our experiments we find that while unargmaxable tokens arise in limited cases, they are not of grave importance. In order to quantify whether unargmaxable classes arise in released LMs and MT models, we first need to introduce tractable algorithms for detecting them. In this Section we explain how unargmaxable classes can arise due to a Softmax Bottleneck. Then, we introduce a fast approximate algorithm and a slow exact algorithm which we combine to detect vocabulary tokens that cannot be predicted. A Softmax layer gives us the probability assigned to a target class c t for an input feature vector x  R d as follows: (1) (2) where W  R | C | d are the class weight vectors stacked row by row, and b  R | C | is the bias term. The above are used to compute the logits y = Wx + b . In what follows, we will refer to the feature activations x in R d as the input space and the logits y in R | C | as the output space of the Softmax layer. As we saw in Figure 2, there are certain arrangements of Softmax weights for which a target class c t cannot be surfaced as the argmax. To understand this phenomenon, it will be helpful to discretise the outputs to a finer granularity: rankings (Burges et al., 2005). In order for a classifier to predict a class c t using an argmax decision rule, it must rank c t above all other classes by assigning it the largest probability. From this perspective, a classifier assigns each input x a permutation  that ranks the class indices in increasing order of probability. As an example, if we have 4 classes and obtain probabilities P ( C | x ) = (cid:2) . 2 . 4 . 1 . 3 (cid:3)  we assign x the permutation  3142 , since P ( c 3 | x ) < P ( c 1 | x ) < P ( c 4 | x ) < P ( c 2 | x ) . We can readily obtain the coarser argmax decision ( c 2 ) by reading off the last index of the permutation. A class c t is unargmaxable when all permutations that rank c t above the rest cannot be realised due to rank constraints. We explain how this happens by combining the following two observations. Observation 1. We can discretise R | C | into regions corresponding to permutations by segmenting the space with hyperplanes. The hyperplanes that partition the output space into regions R  corresponding to permutations are a well known structure in Combinatorics, the Braid 6740 Observation (1): Discretise R | C | into permutations R  123 R  321 R  312 R  132 R  213 R  231 | C | = 3 Observation (2): Observe rank constraints Feasible logits (1) & (2) =  Corollary 1: Feasible permutations R  312 R  321 R  231 R  213 R  123 R  132 R  1342 R  1324 | C | = 4 R  1342 R  1432 R  1423 R  1243 R  2143 R  2413 R  2431 R  2341 R  3241 R  3421 R  3412 R  3142 Figure 3: Illustration of Corollary 1 ( 3 rd column) as a result of Observation 1 ( 1 st column) and Observation 2 ( 2 nd column) for softmax( Wx ) , W  R | C | d , d = 2 . 5 The Braid Arrangement for 3 and 4 classes is illustrated in rows 1 and 2 of Figure 3 respectively. In order to be able to rank the classes according to permutation R  , our network needs to be able to map an input x to region R  in the output space. However, this is not always possible when we have a Softmax Bottleneck as we elaborate below. Case i) softmax( Wx ) . By calculating y = Wx , the class logits y are a linear combination of d columns of W . Therefore, when d < | C | we can only represent a d -dimensional subspace of R | C | at best. This feasible subspace is illustrated as a grey plane in the middle column of Figure 3. Case ii) softmax( Wx + b ) . If we also have a bias term b the model can choose how to offset the subspace. When the bias term b is not in the 5 See Appendix B for more details on hyperplane arrangements and the Braid Arrangement specifically. column space of W the zero vector 0 is no longer a feasible y and instead of a linear subspace we have an affine subspace. See Figure 7 in the Appendix for an illustration comparing the two cases. Corollary 1. A Softmax classifier parametrised by W and b can rank classes in the order of permutation  iff the affine subspace spanned by W and b intersects region R  of the Braid Arrangement . 6 When d < | C |  1 there are regions that cannot be intersected. 7 The feasible permutations in our example correspond to the regions formed on the grey plane illustrated in the rightmost column of Figure 3. Note that for | C | = 4 only 12 out of 24 regions can be intersected. As we make the Softmax Bottleneck narrower by reducing the dimension d of the Softmax inputs, more permutations become infeasible (Good and 6 This insight of slicing the Braid Arrangement was introduced in Kamiya et al. (2011). 7 When d = C  1 we can still intersect all regions, because the Braid Arrangement always has rank | C | 1 (all its normal vectors are perpendicular to the all ones vector 1 ). Tideman, 1977; Kamiya and Takemura, 2005). Importantly, if we choose | C | and d and whether to use a bias term, changing the values of the Softmax weights changes the set of feasible permutations but not the cardinality of the set (Cover, 1967; Smith, 2014). See Appendix C for more details. Corollary 2. Class c t is unargmaxable when any permutation that would rank class c t above all other classes is infeasible. Without a bias term the regions corresponding to permutations are unbounded (see the rightmost column of Figure 3). As such, imposing any range restrictions on the Softmax layer inputs x does not change the feasible regions as long as the restriction includes the origin. However, when we introduce a bias term we also get bounded regions (see Figure 7 in the Appendix that contrasts the two situations). Therefore, in this case the scale of the inputs to the Softmax layer also matters. If the inputs do not have a large enough range, there will be regions that exist but cannot be reached by the feature encoder. Given a softmax layer parametrised by W and b , are there any classes that are unargmaxable? We first describe a slow, but exact algorithm to answer this question. An exact algorithm will either prove class c t is argmaxable by returning a feasible point x : argmax ( Wx + b ) = c t or it will prove c t is unargmaxable by verifying no such point exists. To check if a region exists that ranks c t above all others, we need to find an input x  R d that satisfies the following constraints: P ( c i | x ) < P ( c t | x ) ,  i : 1  i  | C | , i  = t (4) Each of the above constraints is equivalent to restricting x to a halfspace (see Appendix A). Hence, if all above inequalities are enforced, x is restricted to an intersection of halfspaces. If the intersection of halfspaces is empty, there is no x for which class c t can be ranked above all others and hence c t is unargmaxable. We can find a point in an intersection of halfspaces via linear programming, albeit we found this algorithm to be slow in practice for | C | > 1000 . The Chebyshev center of a polytope (Boyd et al., 2004, p. 417) is the center of the largest ball of radius r that can be embedded within the polytope. We can find the Chebyshev center x and the radius r with the following linear programme. maximise r subject to w  i x + r  w i  2   b i , 1  i | C | 1 x  100 x   100 r > 0 (6) Where w i = w c i  w c t and b i = b c i  b c t ,  i : c i  = c t . We further constrain x to guarantee the regions are bounded, since the Chebyshev center is not defined otherwise. This constraint also captures the fact that neural network activations are not arbitrarily large. If the above linear programme is feasible, we know that class c t is argmaxable and we also get a lower bound on the volume of the region for which it is solvable by inspecting r . On the other hand, if the linear programme is infeasible, c t is unargmaxable. The exact algorithm was too slow to run for the whole vocabulary. In order to avoid running the exact algorithm for every single vocabulary item, we developed an incomplete algorithm (Kautz et al., 2009) with a one-sided error, which can quickly rule out most tokens, leaving only a small number to be checked by the exact algorithm. It proves that c t is argmaxable by finding an input x for which c t has the largest activation. Unlike the exact algorithm, if no solution exists it cannot prove that the token is unargmaxable . Hence, we terminate our search after a predetermined number of steps. We denote any tokens not shown to be argmaxable by the approximate algorithm as potentially unargmaxable and we run the exact algorithm on them. An illustration of the way we combine the exact and approximate algorithms to decide whether class c t is argmaxable can be seen in Figure 4. to guide us towards a point x for which c t has the largest activation. To show that class c t is argmaxable, it suffices to find an input x for which the largest probability is assigned to c t . Empirically we found this to be easy for most classes. We begin by interpreting the actual weight vector as the candidate input x = w  c t . We do so since the dot product of two vectors is larger when the two vectors point in the same direction. 8 While the magnitude of the vectors affects the dot product, we found the above initialisation worked well empirically. When c t is not the argmax for x and c i is instead, Relation 5 for c i and c t will have the wrong sign. The sign of this relation defines which side of the Braid hyperplane for c i and c t we are on. To correct the sign, we construct the normal vector and offset of the Braid hyperplane (Lines 2, 3 in Figure 5), compute the distance of x from it (Line 5), and reflect x across it (Line 6). 9 We repeat the above operation until either c t is the argmax or we have used up our budget of N steps. In this Section we use the combined algorithm from Figure 4 to search models for unargmaxable tokens. We test 7 LMs and 143 MT models. We find that unargmaxable tokens only occur in 13 MT models, but these are mostly infrequent and noisy vocabulary tokens. We therefore do not expect such tokens to affect translation quality per se. We also find that nearly all vocabulary tokens of LMs and student MT models can be verified with less than N = 10 steps of the approximate algorithm. In contrast, other MT models need thousands of steps and also rely on the exact algorithm. In this sense, models that need fewer steps are easier to verify: the search problem for their arrangement of Softmax weights is easier. Throughout the following experiments we assumed the Softmax inputs were bounded in magnitude for all dimensions  100  x i  100 . As we mentioned in Subsection 3.2.1, if we have a Softmax bias term, there are bounded regions. If the bounded regions are large, even though the outputs are not theoretically bounded, they are practically bounded since neural network feature encoders cannot produce arbitrarily large activations and some regions may be unreachable 10 . For the approximate algorithm, we search for a solution with a patience of N = 2500 steps and resort to the exact algorithm if the approximate method fails or returns a point outside the aforementioned bounds. We use Gurobi (Gurobi Optimization, 2021) as the linear programme solver. We accessed the model parameters either via NumPy (Harris et al., 2020) or PyTorch (Paszke et al., 2019). The experiments took 3 days to run on an AMD 3900X 12 -core CPU using 10 threads and 64Gb of RAM. We checked 7 widely used LMs for unargmaxable tokens. While some of these models such as 9 When no offset is involved, the reflection operation is the Householder transformation (Householder, 1958). 10 The validity of our assumption is only relevant for models we find to be bounded. BERT (Devlin et al., 2019) are not directly used for generation, a recent trend is to use these large LMs as prompt models (Liu et al., 2021) for few shot learning. A prompt model obviates the need for a separate classifier by rephrasing a classification task as slot filling given a task specific template. Prompt approaches commonly choose the answer for the slot by argmaxing the Softmax distribution obtained by a LM. Hence we verify that there are no answers that are unargmaxable. BERT, RoBERTa (Liu et al., 2019), XLM-RoBERTa (Conneau et al., 2020) and GPT2 (Rad-ford et al., 2019) did not exhibit any unargmaxable tokens and can be assessed without resorting to the exact algorithm (see Table 4 in the Appendix). Moreover, the LMs were very easy to verify with the approximate algorithm requiring less than 1 . 2 steps per token on average. Machine Translation (13/143 Unargmaxable) In the case of MT models, the feature encoder comprises the whole encoder-decoder network excluding the last layer of the decoder. We first focus on models which we found to have unargmaxable tokens and then briefly describe models that did not. A summary of the results and characteristics of the models we checked can be seen in Table 1. More detailed results can be found in Tables 5, 6, 7 and 8 in the Appendix. Helsinki NLP OPUS (13/32 Unargmaxable). The 32 models we use for this subset of experiments are MT models released through Hugging Face (Wolf et al., 2020). We use models introduced in Tiedemann and Thottingal (2020). These models are trained on subsets of OPUS. All models are transformer models trained using Marian (Junczys-Dowmunt et al., 2018). They include a bias term, have a tied encoder and decoder and d = 512 . Unargmaxable tokens, if present, will affect generation in the target language. We therefore restrict our analysis to the target language vocabulary. To 11 https://github.com/browsermt/students facilitate this, we inspect translation models for which the source and target languages have different scripts. We explore 32 models with source and target pairs amongst Arabic (ar), Hebrew (he), English (en), German (de), French(fr), Spanish (es), Finnish (fi), Polish (pl), Greek (el), Russian (ru), Bulgarian (bg), Korean (ko) and Japanese (ja). We rely on the script to disambiguate between source and target language and discard irrelevant tokens from other languages. We also ignore vocabulary tokens containing digits and punctuation. In Figure 6 we can see the number of Byte Pair Encoding (BPE; Sennrich et al., 2016) tokens that were unargmaxable for these models, sorted in decreasing order. As can be seen, all tokens are argmaxable for 19 / 32 language pairs. For the remaining 13 languages, while there can be quite a few unargmaxable tokens, most would not be expected to affect translation quality. Out of the set of 427 unique unargmaxable BPE tokens, 307 / 476 are single character subword tokens and only 2 are word stem BPE segments: erecti (bg-en) and  (en-ru) which means preliminary in Russian. The rest include the <unk> token and noisy subword unicode tokens such as    ,  and  . On closer inspection of the SentencePiece to-keniser we found that both  and erecti come up as tokenisation alternatives that make them rare and irregular. We found that the  token was rare since it is capitalised and only occurs once, while an-other occurrence was caused by a BPE segmentation corner case due to Unicode token variation of -e . Other mentions having  as a substring were split differently. In a similar vein, we found that the erecti token occurred due to BPE corner cases for erecti-0-n , erecti-lis-) , erecti-l , erecti-. and erecti-cle many of which are misspellings or rare word forms from clinical text. As such, the impact of these tokens being unargmaxable is small since there are alternative ones the MT model can prefer over them which could even correct spelling mistakes. FAIR WMT'19 (0/4 Unargmaxable). We checked 4 FAIR models (en-ru, ru-en, en-de, de-en) submitted to WMT'19 (Ng et al., 2019). These transformer models have d = 1024 and do not employ a Softmax bias term. Edinburgh WMT'17 (0/82 Unargmaxable). These WMT'17 submissions (Sennrich et al., 2017) were ensembles of left-to-right trained models (l2r) and right-to-left trained models (r2l). These were LSTMs trained with Nematus using d = 500 or d = 512 and Softmax weights tied with the decoder input embeddings. The models include a bias term. None of the models have unargmaxable tokens. However, we found that models that comprise an ensemble varied a lot in how easy it was to show that the vocabulary was argmaxable, despite them differing solely in the random seed used for weight initialisation. As an example, zh-en.l2r(1) had 8 tokens that needed to be verified with the exact algorithm, zh-en.l2r(2) had 3 and zh-en.l2r(3) had 366 . This highlights that random initialisation alone is enough to lead to very different arrangements of Softmax weights. Bergamot (0/25 Unargmaxable). The Bergamot project 12 model repository contains both large transformer-base and transformer-big teacher models, as well as small knowledge distilled (Kim and Rush, 2016) student models. Student models have d = 256 (tiny) or d = 512 (base), while teacher models have d = 1024 . Interestingly, we find that it is easier to show that student models are argmaxable when compared to teacher models, despite student models having Softmax weights 1 / 2 or 1 / 4 the dimensions of the teacher model. We conclude from our experiments that it is possible to have unargmaxable tokens, but this rarely occurs in practice for tokens that would lead to irrecoverable errors in the MT models we checked. A limitation of our conclusions is that beam search is usually preferred over greedy decoding for MT 12 https://browser.mt models used in practice. We leave the question of whether unargmaxable tokens also impact beam search for future work. It is challenging to make exact claims about what can cause tokens to be unargmaxable because the models we tested varied in so many ways. However, we outline some general trends below. The most general observation is that the tokens that are more likely to be unargmaxable or are hard to prove to be argmaxable are the infrequent ones. This can be seen in Figures 11 and 12 in the Appendix, where the x-axis contains the vocabulary of the models sorted left to right by increasing frequency. Each dot represents the number of steps needed to check whether a token is argmaxable or not, and as can be seen the values to the right are generally much higher than those to the left. This result is in line with previous work that highlights the limitations of the Softmax layer when modelling rare words for LM (Chen et al., 2016; Labeau and Cohen, 2019) and MT (Nguyen and Chiang, 2018; Raunak et al., 2020) and infrequent classes for image classification (Kang et al., 2020). We found that the LMs and student MT model vocabularies can be shown to be argmaxable with one step of the approximate algorithm on average. On the other hand, for Helsinki NLP and FAIR MT models more than 10 steps were needed. To put the above observations into context, we also check the behaviour of our algorithms on randomly initialised parameters. If we initialise a Softmax layer of | C | = 10000 classes using a uniform distribution U (  1 , 1) we do not expect unargmaxable tokens to exist after d = 30 (see Figure 10 in the Appendix). Moreover, any randomly initialised parameters can be checked using the approximate algorithm with fewer steps as we increase d . From this perspective, it is surprising that student models were easier to show to be argmaxable than the teacher models, despite the Softmax weight dimensionality of the student models being much lower (256 for tiny, versus 1024 for teacher). This shows that effective neural MT models do not need to be hard to check, but nevertheless neural models trained on the original data can sometimes converge to such an arrangement of weights. In this work we discretised the outputs of Softmax and showed how dimensionality constraints shrink the set of feasible class rankings and can lead to some classes being impossible to predict using argmax. In our experiments we demonstrated that while MT models can have unargmaxable vocabulary tokens, this does not occur often in our experiments. Moreover, for the models we tested the unargmaxable tokens would not create discernible differences in translation quality as the tokens are noisy and infrequent. We release an algorithm to detect whether some classes are unargmaxable with the hope that this will be helpful to the wider community working on a plethora of different models where the observed phenomena may vary. In future work, we aim to investigate any learn-ability consequences more closely. As we saw, when using an approximate search algorithm, it is much harder to find argmaxable classes in some models than it is in others. Since gradient descent algorithms are also iterative search algorithms seeking optimal parameters, we hypothesise that it will be challenging to train neural network encoders to map activations to regions of the input space that a search algorithm cannot find easily. Hence, although some tokens may not be provably unargmaxable because of constraints imposed by the Softmax parameters of the last layer, some tokens may still be very hard to produce because of difficulties encountered by the feature encoder. To this end, a more holistic investigation into the consequences of the loss in expressivity in low-rank classifiers is warranted. Unargmaxability directly impacts fairness, since certain model outputs, further from being underrepresented, may not be represented at all. As we discussed, low-rank classifiers have limited expressivity compared to full rank classifiers, and thus have to explicitly choose which rankings of classes to retain feasible when using argmax prediction. As such, by choosing to use a low-rank model, we are allowing the data and training procedure to specify which rankings should remain feasible, and harmful biases in our data can be propagated and further exacerbated (Hooker, 2021) by our models due to unargmaxability. For example, it could be the case that underrepresented groups find no representation in the outputs of such models, in the extreme case where related outputs are unargmaxable. As researchers, we should be aware of this limitation when choosing how to parametrise our models (Hooker et al., 2019) and actively seek to either control such phenomena or verify models are not harmful before moving them from research into production. In addition to the above considerations, linear classification layers are vulnerable to targeted attacks via data poisoning techniques (Goldblum et al., 2020), especially under the scenario where shared models are used as feature extractors (Ji et al., 2018). A subset of such techniques, known as feature collisions (Shafahi et al., 2018; Gold-blum et al., 2020), exploit the arrangement of the training examples in feature space to force the misclassification of a target example. Attacks such as Convex Polytope (Zhu et al., 2019) and Bullseye Polytope (Aghakhani et al., 2021), specifically target the unargmaxability weakness (Cover, 1967; Demeter et al., 2020) we elaborated on in the paper. While such attacks assume they are able to inject examples into a training set used for fine-tuning, this is not an unrealistic assumption. This is especially true for recommender systems, where adversarial attacks can create fake users such that a target item is removed from a target user's top-k list (Christakopoulou and Banerjee, 2019). We thank Seraphina Goldfarb-Tarrant, Elizabeth Nielsen and Sabine Weber for help with languages, Beatrice Alex, Sameer Bansal, Panagiotis Eu-stratiadis, Sharon Goldwater, Chantriolnt-Andreas Kapourani, Oli Liu, Yevgen Matusevych, Kate McCurdy, Laura Perez-Beltrachini, Jesse Sigal, Mark Steedman, Ivan Titov and Sabine Weber for feedback and support, Antonio Vergari for feedback, guidance and tirelessly discussing low-rank constraints and Shay Cohen for insightful suggestions and for pointing us to OEIS. We also thank David Demeter for an extensive discussion on Stolen Probability and the anonymous reviewers for helpful questions and comments. This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/R513209/1] and Research and Innovation Action Bergamot , which has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825303.
abstain abstain abstain abstain abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method objective abstain abstain abstain abstain objective objective abstain result abstain objective abstain abstain abstain abstain abstain abstain method abstain abstain abstain result method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain result method abstain abstain result objective method abstain objective result result abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain other other other