Distributional semantic models have become a mainstay in NLP, providing useful features for downstream tasks. However, assessing long-term progress requires explicit long-term goals. In this paper, I take a broad linguistic perspective, looking at how well current models can deal with various semantic challenges. Given stark differences between models proposed in different subfields, a broad perspective is needed to see how we could integrate them. I conclude that, while linguistic insights can guide the design of model architectures, future progress will require balancing the often conflicting demands of linguistic expressiveness and computational tractability. In order to assess progress in any field, the goals need to be clear. In assessing progress in semantics, Koller (2016) contrasts top-down and bottom-up approaches: a top-down approach begins with an overarching goal, and tries to build a model to reach it; a bottom-up approach begins with existing models, and tries to extend them towards new goals. 1 Like much of NLP, distributional semantics is largely bottom-up: the goals are usually to improve performance on particular tasks, or particular datasets. Aiming to improve NLP applications is of course a legitimate decision, but Koller points out a problem if there is no top-down goal: Bottom-up theories are intrinsically unfalsifiable... We won't know where distributional semantics is going until it has a top-down element. This is contrasted against truth-conditional semantics, a traditional linguistic approach which is largely top-down: truth-conditional semantics hasn't reached its goal, but at least we knew what the goal was. 1 For further discussion, see: Bender and Koller (2020). the meanings of all utterances in a language. This is an ambitious goal, and a broad one. To make this goal more precise, in the following sections I will elaborate on several aspects of meaning which could be considered crucial. For each aspect, I identify a plausible goal, lay out out the space of possible models, place existing work in this space, and evaluate which approaches seem most promising. By making the goals explicit, we can assess whether we are heading in the right direction, and we can assess what still needs to be done. If a reader should disagree with my conclusions, they should start by looking at my goals. The aim of distributional semantics is to learn the meanings of linguistic expressions from a corpus of text. The core idea, known as the distributional hypothesis , is that the contexts in which an expression appears give us information about its meaning. 2 The idea has roots in American structuralism (Harris, 1954) and British lexicology (Firth, 1951, 1957) 3 , and with the advent of modern computing, it began to be used in practice. In a notable early work, Sparck-Jones (1964) represented word meanings as boolean vectors, based on a thesaurus. Distributional semantics has become widespread in NLP, first with the rise of count vectors (for an overview, see: Erk, 2012; Clark, 2015), then of word embeddings (Mikolov et al., 2013), and most recently, of contextualised embeddings (Pe-ters et al., 2018; Devlin et al., 2019). 4 What all of these approaches share is that they learn representations in an unsupervised manner on a corpus. 2 The hypothesis is often stated more narrowly, to say that similar words appear in similar contexts, but in this paper I am interested in semantics beyond just similarity. 3 Firth used the term collocational , not distributional . 4 For connections between count vectors and embeddings, see: Levy and Goldberg (2014); Cotterell et al. (2017); for connections with contextual embeddings: Kong et al. (2020). While much work takes a bottom-up approach, as Koller observes, a notable exception is the type-driven tensorial framework of Coecke et al. (2010) and Baroni et al. (2014), which has broad linguistic goals, and will be mentioned in several sections below. This framework represents the meanings of words as tensors, and constructs phrase meanings using tensor contraction based on predicate-argument structure. For example, there is one vector space for nouns, and a second vector space for sentences, so intransitive verbs are matrices (map-ping noun vectors to sentence vectors). Language is always about something. In this section, I discuss challenges in connecting a semantic model to things in the world. As Harnad (1990) discusses, if the meanings of words are defined only in terms of other words, these definitions are circular. One goal for a semantic model is to capture how language relates to the world, including sensory perception and motor control  this process of connecting language to the world is called grounding . 5 A purely distributional model is not grounded, as it is only trained on text, with no direct link to the world. There are several ways we could try to ground a distributional model (for an overview, see: Baroni, 2016). The simplest way is to train a distributional model as normal, then combine it with a grounded model. For example, Bruni et al. (2011) concatenate distributional vectors and image feature vectors. This has also been applied to other senses: Kiela et al. (2015) use olfactory data, and Kiela and Clark (2017) use both visual and auditory data. However, while there is grounded information in the sensory dimensions, concatenation leaves the distributional dimensions ungrounded. A second approach is to find correlations between distributional and sensory features. For example, Bruni et al. (2014) perform SVD on concatenated vectors, Silberer and Lapata (2014) train an autoencoder on concatenated vectors, and Lazaridou et al. (2014) and Bulat et al. (2016) learn a mapping from distributional vectors to visual vectors (and vice versa). However, there is no guarantee 5 This includes connecting abstract concepts to the world, although such connections are necessarily more indirect. For further discussion, see: Blondin-Mass  e et al. (2008); Pecher et al. (2011); Pulverm  uller (2013); Barsalou et al. (2018) that every distributional feature will correlate with sensory features. Distributional features without correlations will remain ungrounded. Finally, a third approach is joint learning  we define a single model, whose parameters are learnt based on both corpus data and grounded data. For example, Feng and Lapata (2010) train an LDA model (Blei et al., 2003) for both words and visual words (clusters of visual features). Lazaridou et al. (2015) use a Skip-gram model (Mikolov et al., 2013) to jointly predict both words and images. Kiros et al. (2014) embed both text and images in a single space, training an RNN to process captions, and a CNN to process images. Pure distributional models look for word co-occurrence patterns, while joint models prefer co-occurrence patterns that match the grounded data. For this reason, I believe joint learning is the right approach to ground corpus data  semantic representations can be connected to grounded data from the outset, rather than trying to make such connections after the fact. However, we must still make sure that all distributional features are grounded. With Feng and Lap-ata's LDA model, some topics might only generate words rather than visual words. Similarly, with Lazaridou et al.'s joint Skip-gram model, some embeddings might only predict words rather than images. Conversely, we also need to make sure that we make full use of corpus data, rather than discarding what is difficult to ground. For example, Kiros et al.'s joint embedding model learns sentence embeddings in order to match them to images. It is not obvious how this approach could be extended so that we can learn embeddings for sentences that cannot be easily depicted in an image. This leads to the question: how should a joint architecture be designed, so that we can fully learn from corpus data, while ensuring that representations are fully grounded? Grounding is hard, and indeed Kuhnle et al. (2018) find that some semantic constructions (such as superlatives) are much harder for grounded models to learn than others. In the following section, I discuss how language relates to the world. Clarifying this relationship should help us to design good joint architectures. How do meanings relate to the world? In truth-conditional semantics, the answer is that meaning is defined in terms of truth . 6 If an agent under-6 For a discussion of this point, see: Lewis (1970). For an stands a language, then in any given situation, they know how to evaluate whether a sentence is true or false of that situation. 7 An advantage of this approach is that it supports logical reasoning, which I will discuss in  5.2. One goal for a semantic theory is to be able to generalise to new situations. This is difficult for traditional truth-conditional semantics, with classical theories challenged on both philosophical grounds (for example: Wittgenstein, 1953,  6671) and empirical grounds (for example: Rosch, 1975, 1978). However, a machine learning approach seems promising, since generalising to new data is a central aim of machine learning. For a semantic model to be compatible with truth-conditional semantics, it is necessary to distinguish a concept (the meaning of a word) from a referent (an entity the word can refer to). 8 The importance of this distinction has been noted for some time (for example: Ogden and Richards, 1923). A concept's set of referents is called its extension . 9 Even if we can construct grounded concept vectors, as discussed in  3.1, there is still the question of how to relate a concept vector to its referents. 10 One option is to embed both concepts and entities in the same space. We then need a way to decide how close the vectors need to be, for the entity to be in the concept's extension. A second option is to embed concepts and referents in distinct spaces. We then need a way to relate the two spaces. In both cases, we need additional structure beyond representing concepts and referents as points. One solution is to represent a concept by a region of space (Gardenfors, 2000, 2014). Entities embedded inside the region are referents, while those outside are not. For example, McMahan and Stone (2015) learn representations of colour terms, which are grounded in a well-understood perceptual space. A related idea is to represent a concept as a binary classifier, where an entity is the input. 11 One class is the concept's extension, and the other class introduction to truth-conditional semantics, see: Cann (1993); Allan (2001); Kamp and Reyle (2013). 7 On the notion of situation , see: Barwise and Perry (1983). On knowing how to evaluate truth values vs. actually evaluating truth values, see: Dummett (1976, 1978). 8 Following Murphy (2002, pp. 45), I use the term concept without committing to a particular theory of concepts. 9 Or denotation . In psychology, the term category is also used (for example: Smith and Medin, 1981; Murphy, 2002). 10 While distributional representations can be learnt for named entities (for example: Herbelot, 2015; Boleda et al., 2017), most real-world entities are not mentioned in text. 11 For deterministic regions and classifiers, there is a one-toone mapping between them, but this is not true for probabilistic regions and classifiers, due to covariance. is everything else. Larsson (2013) represents the meaning of a perceptual concept as a classifier of perceptual input. A number of authors have trained image classifiers using captioned images (for example: Schlangen et al., 2016; Zarrie and Schlangen, 2017a,b; Utescher, 2019; Matsson et al., 2019). Such representations have however seen limited use in distributional semantics. Erk (2009a,b) and Dong et al. (2018) learn regions, but relying on pre-trained vectors, which may have already lost referential information (such as co-reference) that we would like to capture. Jameel and Schockaert (2017) learn a hybrid model, where each word is represented by a point (as a target word) and a region (as a context word). In my own work, I have learnt classifiers (Emerson and Copestake, 2016, 2017a,b), but with a computationally expensive model that is difficult to train. The computational challenge is partially resolved in my most recent work (Emerson, 2020a), but there is still work to be done in scaling up the model to make full use of the corpus data. The best way to design such a model, so that it can both make full use of the data and can be trained efficiently, is an open question. In this section, I discuss challenges in representing the meanings of individual words. Entities often fall along a continuum without a sharp cutoff between concepts. This is called vagueness (or gradedness ). (For an overview, see: Sutton, 2013, chapter 1; Van Deemter, 2010.) For example, Labov (1973) investigated the boundaries between concepts like cup , mug , and bowl , asking participants to name drawings of objects. For typical referents, terms were used consistently; meanwhile, for objects that were intermediate between concepts (for example, something wide for a cup but narrow for a bowl), terms were used inconsistently. For these borderline cases, a single person may make different judgements at different times (McCloskey and Glucksberg, 1978). One goal for a semantic model is to capture how it can be unclear whether an entity is an referent of a concept. One approach is to use fuzzy truth values, which are not binary true/false, but rather values in the range [0,1], where 0 is definitely false, 1 is definitely true, and intermediate values represent borderline cases (Zadeh, 1965, 1975). Fuzzy logic has not seen much use in computational linguistics. 12 A second solution is to stick with binary truth values, but using probability theory to formalise uncertainty about truth, as has been proposed in formal semantics (for example: Lassiter, 2011; Fernandez and Larsson, 2014; Sutton, 2015, 2017). At the level of a single concept, there is not much to decide between fuzzy and probabilistic accounts, since both assign values in the range [0,1]. However, we will see in  5.2 that they behave differently at the level of sentences. Uncertainty has also been incorporated into distributional vector space models. Vilnis and McCallum (2015) extend Mikolov et al.'s Skip-gram model, representing meanings as Gaussian distributions over vectors. Barkan (2017) incorporate uncertainty into Skip-gram using Bayesian inference  rather than optimising word vectors, the aim is to calculate the posterior distribution over word vectors, given the observed data. The posterior is approximated as a Gaussian, so these two approaches produce the same kind of object. Balkr (2014), working within the type-driven tensorial framework (see  2), uses a quantum mechanical mixed state to model uncertainty in a tensor. For example, this replaces vectors by matrices, and replaces matrices by fourth-order tensors. While these approaches represent uncertainty, it is challenging to use them to capture vagueness. This basic problem is this: a distribution allows us to generate referents of a concept, but how can we go in the other direction, to recognise referents of a concept? It is tempting to classify a point using the probability density at that point, but if we compare a more general term with a more specific term (like animal and dog ), we find a problem: a more general term has its probability mass spread more thinly, and hence has a lower probability density than the more specific term, even if both terms could be considered true. I argued in  3.2 that, to talk about truth, we need to represent predicates as regions of space or as classifiers. While a distribution over a space might at first sight look like a region of space, normalising the probability mass to sum to 1 makes a distribution a different kind of object. 12 Carvalho et al. (2012) survey fuzzy logic in NLP, noting that its use is in decline, but they do not mention distributional semantics. Proposals such as Monte Carlo Semantics (Bergmair, 2010) and Fuzzy Natural Logic (Nov  ak, 2017) do not provide an approach to distributional semantics. A rare exception is Runkler (2016), who infers fuzzy membership functions from pre-trained vectors. The meaning of a word can often be broken up into distinct senses . Related senses are called polysemous : for example, school can refer to a building or an institution. In contrast, homonymous senses are unrelated: for example, a school of fish. All of the above senses of school are also lexicalised  established uses that a speaker would have committed to memory, rather than inferring from context. I will discuss context-dependent meaning in  5.3, and focus here on lexicalised meaning. One goal for a semantic model is to capture how a word can have a range of polysemous senses. One solution is to learn a separate representation for each sense (for example: Schutze, 1998; Rapp, 2004; Li and Jurafsky, 2015; for a survey, see: Camacho-Collados and Pilehvar, 2018). However, deciding on a discrete set of senses is difficult, and practical efforts at compiling dictionaries have not provided a solution. Indeed, the lexicographer Sue Atkins bluntly stated, I don't believe in word senses. 13 Although the sense of a word varies across usages, there are many ways that we could cluster usages into a discrete set of senses, a point made by many authors (for example: Sparck-Jones, 1964; Kilgarriff, 1997, 2007; Hanks, 2000; Erk, 2010). To quantify this intuition, Erk et al. (2009, 2013) produced the WSsim and Usim datasets, where annotators judged the similarity between dictionary senses, and the similarity between individual usages, respectively. McCarthy et al. (2016) quantify clusterability in USim, showing that for some words, usages cannot be clustered into discrete senses. A good semantic model should therefore be able to capture variation in meaning without resorting to finite sense inventories. We could instead learn a single representation for all polysemous senses together. Indeed, Ruhl (1989) argues that even frequent terms with many apparent senses, such as bear and hit , can be analysed as having a single underspecified meaning, with the apparent diversity of senses explainable from context. The challenge is then to represent such a meaning without overgeneralising to cases where the word wouldn't be used, and to model how meanings are specialised in context. The second half of this challenge will be discussed in  5.3. I have already argued in previous sections that we should move away from representing each word as a single vector. As discussed in  4.1, words 13 Kilgarriff (1997) and Hanks (2000) both quote Atkins. can be represented with distributions, and such an approach has also been applied to modelling word senses. For example, Athiwaratkun and Wilson (2017) use a mixture of Gaussians, extending Vilnis and McCallum's model to allow multiple senses. However, this ultimately models a fixed number of senses (one for each Gaussian). In principle, a distribution could be parametrised in a more general way, moving beyond finite mixture models. In the type-driven tensorial framework (see  2), Piedeleu et al. (2015) use mixed quantum states, similarly to Balkr's approach (see  4.1). Although they only propose this approach for homonymy, it could plausibly be used for polysemy as well. If a word is represented by a region, or by a classifier, we don't have the problem of finite sense inventories, as long as the region or classifier is parametrised in a general enough way  for example, a multi-layer neural net classifier, rather than a finite mixture of simple classifiers. In the previous two sections, I discussed meanings of single words. However, words do not exist on their own, and one goal for semantic model is to represent relations between them. A classic relation is hyponymy , 14 which describes when one term (the hyperonym or hypernym ) has a more general meaning than another (the hyponym ). Words that share a hyperonym are called co-hyponyms . In a vector space model, it is not clear how to say if one vector is more general than another. One idea is that a hyperonym should occur in all the contexts of its hyponyms. This is known as the Distributional Inclusion Hypothesis (DIH; Weeds et al., 2004; Geffet and Dagan, 2005). Using this idea and tools from information retrieval, Kotlerman et al. (2009, 2010) define the balAPinc measure of hyponymy. Herbelot and Ganesalingam (2013) view a vector as a distribution over contexts, using KL-divergence to measure hyponymy. Rei (2013) gives an overview of hyponymy measures, and proposes a weighted cosine measure. For embeddings, the motivation for such measures is less direct, but dimensions can be seen as combinations of contexts. Indeed, Rei and Briscoe (2014) find embeddings perform almost as well as count vectors. 14 This is also referred to as lexical entailment , making a link with logic (see  5.2). Other relations include antonymy, meronymy, and selectional preferences. For reasons of space, I have decided to discuss one relation in detail, rather than many relations briefly. Hyponymy could be considered basic. However, a speaker is likely to choose an expression with a degree of generality appropriate for the discourse (the Maxim of Quantity; Grice, 1967), and hence the DIH can be questioned. Rimell (2014) points out that some contexts are highly specific. For example, mane is a likely context of lion but not animal , even though lion is a hyponym of animal , contradicting the DIH. Rimell instead proposes measuring hyponymy using coherence (formalised using pointwise mutual information): the contexts of a general term minus those of a hyponym are coherent, but the reverse is not true. Moving away from count vectors and pre-trained embeddings, there are other options. One is to build the hyponymy relation into the definition of the space. For example, Vendrov et al. (2016) use nonnegative vectors, where one vector is a hyponym of another if it has a larger value in every dimension. They train a model on WordNet (Miller, 1995; Fellbaum, 1998). Building on this, Li et al. (2017) learn from both WordNet and text. However, for a hierarchy like WordNet, there are exponentially more words lower down. This cannot be embedded in Euclidean space without words lower in the hierarchy being increasingly close together. Nickel and Kiela (2017) propose using hyperbolic space, where volume increases exponentially as we move away from any point. T , ifrea et al. (2019) build on this, adapting Glove (Pennington et al., 2014) to learn hyperbolic embeddings from text. However, this approach does not generalise to non-tree hierarchies  for example, WordNet gives bass as a hyponym of singer , voice , melody , pitch , and instrument . Requiring that bass is represented close to all its hyperonyms also forces them close together (by the triangle in-equality), which we may not want, since they are in distant parts of the hierarchy. Alternatively, we can view hyponymy as classification, and simply use distributional vectors to provide input features (for example: Weeds et al., 2014; Rei et al., 2018). However, under this view, hyponymy is an opaque relationship, making it difficult to analyse why one vector is classified as a hyponym of another. Indeed, Levy et al. (2015) find that such classifiers mainly learn which words are common hyperonyms. Moving away from vector representations, it can be easier to define hyponymy. Erk (2009a,b) and Gardenfors (2014,  6.4) discuss how using regions of space provides a natural definition: P is a hyponym of Q if the region for P is contained in the region for Q . Bouraoui et al. (2017) and Vilnis et al. (2018) use this idea for knowledge base completion, and Bouraoui et al. (2020) build on this, using corpus data to identify conceptual neighbours. In the type-driven tensorial framework (see  2), Bankova et al. (2019) and Lewis (2019) model words as normalised positive operators, with hyponymy defined in terms of subspaces (eigenspaces). Probability distributions also allow us to define hyponymy, but it is harder than for regions, since a distribution over a smaller region has higher probability density. Vilnis and McCallum (2015) propose using KL-divergence. Athiwaratkun and Wilson (2018) propose a thresholded KL-divergence. In the type-driven tensorial framework, Balkr (2014) proposes using a quantum version of KL-divergence, which can be extended to phrases (Balkr et al., 2015; Sadrzadeh et al., 2018). However, detecting hyponymy from corpus data remains challenging. Even in recent shared tasks (Bordea et al., 2016; Camacho-Collados et al., 2018), many systems use pattern matching, following Hearst (1992). For example, a string of the form X such as Y suggests that Y is a hyponym of X . In the above shared tasks, the best performing systems did not rely solely on distributional vectors, but used pattern matching as well. Although much work remains to be done in developing learning algorithms which can detect hyponymy, I believe that a region-based approach is the most promising. Not only does it give a simple definition, but it is also motivated for other reasons, discussed elsewhere in this paper. In the previous section, I discussed meaning at the level of words. I now turn to challenges in representing meaning at the level of sentences. Language is productive  a fluent speaker can understand a completely new sentence, as long as they know each word and each syntactic construction in the sentence. One goal for a semantic model is to be able to derive the meaning of a sentence from its parts, so it can generalise to new combinations. This is known as compositionality . 15 15 Kartsaklis et al. (2013) discuss how composition is often conflated with disambiguation , since composing ambiguous expressions often disambiguates them. Disambiguation can be seen as a kind of contextualisation or context dependence , For vector space models, the challenge is how to compose word vectors to construct phrase representations. If we represent both words and phrases in the same vector space, the challenge is to find a composition function that maps a pair of vectors to a new vector. In the general case, this must be sensitive to word order, since changing word order can change meaning. Mitchell and Lapata (2008, 2010) compare a variety of such functions, but find that componentwise multiplication performs best, despite being commutative, and hence insensitive to word order. The effectiveness of componentwise multiplication and addition has been replicated many times (for example: Baroni and Zamparelli, 2010; Blacoe and Lapata, 2012; Rimell et al., 2016; Czarnowska et al., 2019). However, it is unclear how to adapt it to take word order into account, and Polajnar et al. (2014) show that performance degrades with sentence length. Alternatively, we can use a sentence space distinct from the word space. This is often done with a task-based perspective  words are combined into sentence representations, which are useful for solving some task. For example, the final state of an RNN can be seen as a representation of the whole sequence. To make the composition more linguistically informed, the network can be defined to follow a tree structure, rather than linear order (for example: Socher et al., 2010, 2012; Tai et al., 2015), or even to learn latent tree structure (for example: Dyer et al., 2016; Maillard and Clark, 2018). Alternatively, a sequence of token representations can be combined using attention, which calculates a weighted sum, as in a Transformer architecture (Vaswani et al., 2017). Regardless of architecture, the model can be optimised either for a supervised task, such as machine translation (for example: Cho et al., 2014), or for an unsupervised objective, as in an autoencoder (for example: Hermann and Blunsom, 2013) or language model (for example: Peters et al., 2018; Devlin et al., 2019). If we take a task-based perspective, it is difficult to know if the representations will transfer to other tasks. In fact, Changpinyo et al. (2018) find that for some combinations of tasks, training on one task can be harmful for another. which I discuss in  5.3. The focus in this section is on deriving semantic representations for larger expressions. compose representations based on argument structure. 16 Polajnar et al. (2015) explore sentence spaces with dimensions defined by co-occurrences. However, a weakness with the above approaches is that they map sentences to a finite-dimensional space. As we increase sentence length, the number of sentences with distinct meanings increases exponentially. For example, consider relative clauses: the dog chased the cat ; the dog chased the cat which caught the mouse ; and so on. To keep these meanings distinct, we have two options. If the meanings must be a certain distance apart, the magnitudes of sentence vectors need to increase exponentially with sentence length, so there is enough space to distinguish them. 17 Alternatively, if the meanings can be arbitrarily close, we need to record each dimension to a high precision in order to distinguish the meanings. The fine-grained structure of the space then becomes important, but small changes to model parameters (such as updates during training) would cause drastic changes to this structure. I do not know any work exploring either option. Otherwise, we are forced to view sentence vectors as lossy compression. 18 As Mooney (2014) put it: You can't cram the meaning of a whole %&!$# sentence into a single $&!#* vector! Although compression can be useful for many tasks, full and detailed semantic representations also have their place. This is particularly important at a discourse level: it would be absurd to represent, as vectors of the same dimensionality, both a five-word sentence and the whole English Wikipedia. However, this leaves open the question of how we should represent sentence meaning. In the following section, I turn to logic as a guide. Sentences can express complex thoughts, and build chains of reasoning. Logic formalises this, and one goal for a semantic model is to support the logical notions of truth (discussed in  3.2), and entailment (one proposition following from another). Vectors do not have logical structure, but can still 16 Zanzotto et al. (2015) show how sentence similarity in this framework decomposes in terms of similarity of corresponding parts, because composition and dot products are linear. 17 This can be formalised information-theoretically. Consider sending a message as a D -dimensional vector, through a noisy channel. If there is an upper bound K to the vector's magnitude, the channel has a finite channel capacity . The capacity scales as KD , which is only polynomial in K . 18 This conclusion has been drawn before (for example: Goodfellow et al., 2016, p. 370), but my argument makes the conditions more precise. be used to provide features for a logical system, for example if entailment is framed as classification: given a premise and hypothesis , the task is to decide if the premise entails the hypothesis, contradicts it, or neither. Datasets include SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018). However, it is difficult to analyse approaches that do not use an explicit logic. In fact, Gururan-gan et al. (2018) suggest that high performance may be due to annotation artifacts: only using the hypothesis, they achieve 67% on SNLI and 53% on MultiNLI, much higher than the majority class baseline (34% and 35%, respectively). Performance on such datasets may therefore overestimate the ability of neural models to perform inference. To explicitly represent logical structure, there are a few options. One is to build a hybrid system, combining a vector space with a logic. For example, Herbelot and Vecchi (2015) aim to give logical interpretations to vectors. They consider a number of properties (such as: is edible , has a handle , made of wood ), and for each, they learn a mapping from vectors to values in [0 , 1] , where 0 means the property applies to no referents, and 1 means it applies to all referents. This is an interesting way to probe what information is available in distributional vectors, but it is unclear how it could be generalised to deal with individual referents (rather than summarising them all), or to deal with complex propositions (rather than single properties). Garrette et al. (2011) and Beltagy et al. (2016) incorporate a vector space model into a Markov Logic Network (Richardson and Domingos, 2006), a kind of probability logic. If two predicates have high distributional similarity, they add a probabilistic inference rule saying that, if one predicate is true of an entity, the other predicate is likely to also be true. This allows us to use distributional vectors in a well-defined logical model, but it assumes we can interpret similarity in terms of inference (for discussion, see: Erk, 2016). As argued in  3 above, pre-trained vectors may have already lost information, and in the long term, it would be preferable to learn logical representations directly. Lewis and Steedman (2013) use a classical logic, and cluster predicates that are observed to hold of the same pairs of named entities  for example, write ( Rowling , Harry Potter ) and author ( Rowling , Harry Potter ). This uses corpus data directly, rather than pre-trained vectors. However, it would need to be generalised to learn from arbitrary sentences, and not just those involving named entities. A second option is to define a vector space with a logical interpretation. Grefenstette (2013) gives a logical interpretation to the type-driven tensorial framework (see  2), where the sentence space models truth values, and the noun space models a domain of N entities. However, Grefenstette shows that quantification would be nonlinear, so cannot be expressed using tensor contraction. Hedges and Sadrzadeh (2019) provide an alternative account which can deal with quantifiers, but at the expense of noun dimensions corresponding to sets of entities, so we have 2 N dimensions for N entities. Copestake and Herbelot (2012) propose that dimensions could correspond to logical expressions being true of an entity in a situation. However, this requires generalising from an actual distribution (based on observed utterances) to an ideal distribution (based on truth of logical expressions). They do not propose a concrete algorithm, but they discuss several challenges, and suggest that grounded data might be necessary. In this vein, Kuzmenko and Herbelot (2019) use the Visual Genome dataset (Krishna et al., 2017) to learn vector representations with logically interpretable dimensions, although these vectors are not as expressive as Copestake and Herbelot's ideal distributions. Finally, a third option is to learn logical representations instead of vectors. For example, in my own work I have represented words as truth-conditional functions that are compatible with first-order logic (Emerson and Copestake, 2017b; Emerson, 2020b). Since referents are not observed in distributional semantics, this introduces latent variables that make the model computationally expensive, although there are ways to mitigate this (Emerson, 2020a). Despite the computational challenges, I believe the right approach is to learn a logically interpretable model, either by defining a vector space with logical structure, or by directly using logical representations. However, an important question is what kind of logic to use. I argued in  4.1 that probabilities of truth and fuzzy truth values can capture vagueness, and there are corresponding logics. In probability logic, propositions have probabilities of being true or false, with a joint distribution for the truth values of all propositions (for an introduction, see: Adams, 1998; Demey et al., 2013). In fuzzy logic, propositions have fuzzy truth values, and classical logical operators (such as:  ,  ,  ) are replaced with fuzzy versions (for an introduction, see: Hajek, 1998; Cintula et al., 2017). Fuzzy operators act directly on truth values  for example, given the fuzzy truth values of p and q , we can calculate the fuzzy truth value of p  q . In contrast, in probability logic, given probabilities of truth for p and q , we cannot calculate the probability of truth for p  q , unless we know the joint distribution. A problem with fuzzy logic, observed by Fine (1975), comes with propositions like p   p . For example, suppose we have a reddish orange object, so the truth of red and orange are both below 1. Intuitively, both red or not red and red or orange should definitely be true. However, in fuzzy logic, they could have truth below 1. This makes probability logic more appealing than fuzzy logic. 19 Furthermore, there are well-developed frameworks for probabilistic logical semantics (for example: Goodman and Lassiter, 2015; Cooper et al., 2015), which a probabilistic distributional semantics could connect to, or draw inspiration from. The flipside of compositionality is context dependence : the meaning of an expression often depends on the context it occurs in. For example, a small elephant is not a small animal , but a large mouse is  the meanings of small and large depend on the nouns they modify. One goal for a semantic model is to capture how meaning depends on context. 20 Following Recanati (2012), we can distinguish standing meaning , the context-independent meaning of an expression, and occasion meaning , the context-dependent meaning of an expression in a particular occasion of use. 21 However, every usage occurs in some context, so a standing meaning must be seen as an abstraction across usages, rather than a usage in a null context (for discussion, see: Searle, 1980; Elman, 2009). One approach is to treat a distributional vector as a standing meaning, and modify it to produce occasion meanings. For example, vectors could be modified according to syntactic or semantic dependencies (for example: Erk and Pado, 2008; Thater et al., 2011; Dinu et al., 2012), or even chains of 19 H  ajek et al. (1995) prove that fuzzy logic can be used to provide upper and lower bounds on probabilities in a probability logic, giving it a different motivation. 20 Ultimately, this must include dependence on real-world context. Even the intuitive conclusion that a large mouse is a small animal depends on the implicit assumption that you and I are both humans, or at least, human-sized. From the perspective of an ant, a mouse is large animal. 21 This terminology adapts Quine (1960). dependencies (for example: Weir et al., 2016). This mapping from standing vectors to occasion vectors can also be trained (for example: Czarnowska et al., 2019; Popa et al., 2019). Large language models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) can also be interpreted like this  these models map a sequence of input embeddings to a sequence of contextualised embeddings, which can be seen as standing meanings and occasion meanings, respectively. Alternatively, standing meanings and occasion meanings can be represented by different kinds of object. Erk and Pado (2010) represent a standing meaning as a set of vectors (each derived from a single sentence of the training corpus), and an occasion meaning is a weighted sum of these vectors. For a probabilistic model, calculating an occasion meaning can be cast as Bayesian inference, conditioning on the context. This gives us a well-understood theoretical framework, making it easier to generalise a model to other kinds of context. Dinu and Lapata (2010) interpret a vector as a distribution over latent senses, where each component is the probability of a sense. Given probabilities of generating context words from latent senses, we can then condition the standing distribution on the context. However this model relies on a finite sense inventory, which I argued against in  4.2. Lui et al. (2012) and Lau et al. (2012, 2014) use LDA (Blei et al., 2003), where an occasion meaning is a distribution over context words (vary-ing continuously as topic mixtures), and a standing meaning is a prior over such distributions. 22 A separate model is trained for each target word. Chang et al. (2014) add a generative layer, allowing them to train a single model for all target words. However, a single sense is chosen in each context, giving a finite sense inventory. Skip-gram can be interpreted as generating context words from a target word. While we can see an embedding as a standing meaning, nothing can be seen as an occasion meaning. Brazinskas et al. (2018) add a generative layer, generating a latent vector from the target word, then generating context words from this vector. We can see a latent vector as an occasion meaning, and a word's distribution over latent vectors as a standing meaning. occasion meanings by conditioning on the context (Emerson and Copestake, 2017b), but in contrast to the above approaches, standing meanings are truth-conditional functions (binary classifiers), which I have argued for elsewhere in this paper. A common thread among all of the above sections is that reaching our semantic goals requires structure beyond representing meaning as a point in space. In particular, it seems desirable to represent the meaning of a word as a region of space or as a classifier, and to work with probability logic. However, there is a trade-off between expressiveness and learnability: the more structure we add, the more difficult it can be to work with our representations. To this end, there are promising neural architectures for working with structured data, such dependency graphs (for example: Marcheggiani and Titov, 2017) or logical propositions (for example: Rocktaschel and Riedel, 2017; Minervini et al., 2018). To mitigate computationally expensive calculations in probabilistic models, there are promising new techniques such as amortised variational inference, used in the Variational Autoencoder (Kingma and Welling, 2014; Rezende et al., 2014; Titsias and Lazaro-Gredilla, 2014). My own recent work in this direction has been to develop the Pixie Autoencoder (Emerson, 2020a), and I look forward to seeing alternative approaches from other authors, as the field of distributional semantics continues to grow. I hope that this survey paper will help other researchers to develop the field in a way that keeps long-term goals in mind. This paper is based on chapter 2 of my PhD thesis (Emerson, 2018). For invaluable advice on the structure and framing of that chapter, and therefore also of this paper, I want to thank my PhD supervisor Ann Copestake. I would also like to thank my PhD examiners, Katrin Erk and Paula Buttery, for feedback on that chapter, as well as Emily M. Bender, Guy Aglionby, Andrew Caines, and the NLIP reading group in Cambridge, for feedback on earlier drafts of this paper. I would like to thank ACL reviewers 1 & 3 for pointing out areas that were unclear, and reviewer 2 for their kind praise. I am supported by a Research Fellowship at Gonville & Caius College, Cambridge.
abstain abstain method objective abstain abstain abstain abstain objective objective abstain abstain abstain abstain abstain objective abstain other abstain other other other other abstain other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain abstain abstain abstain other other other other other