Answers.Body: <p>I can't really make much sense of Eisenstein's distinction between <em>distributional</em> and <em>distributed</em>. And I think in your question you actually mix up the two terms as well, as distribut<strong>ed</strong> semantics involve symbolic structures, whereas distribut<strong>ional</strong> semantics are numerical vectors according to his definition. EDIT: actually, he seems to mix it up himself there?! Very unclear paragraph there.</p>
  <p>I can only imagine that the symbolic structures he refers to here are semantic networks and the like, as in</p>
  <blockquote>
  <p>(is-a feline mammal)<br />
  (is-a lion feline)<br />
  (has-a feline tail)</p>
  </blockquote>
  <p>Here the meaning of <em>lion</em>, as a feline mammal with a tail, is defined through a symbolic structure, and not in reference to the context of usage. Why this should be distributed, I can only guess: the meaning components are split over a set of statements, which build up a larger structure perhaps?</p>
  <p>It could, of course, be the case that this is covered elsewhere in the book — I haven't had the time to look through all of it.</p>
  <p>UPDATE: Thinking more about this, perhaps he means that distributional semantics are representations where each word is a straight co-occurrence vector, ie a vector as large as the words used to define contexts, while distributed semantics is similar, but it's a different vector which is created through processing the contexts (and could thus be smaller)?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-09T12:45:01.600Z
Answers.LastActivityDate: 2021-06-09T12:50:21.373Z
Answers.LastEditDate: 2021-06-09T12:50:21.373Z
Answers.UserDisplayName: Oliver Mason
Answers.Body: <p><em>I am writing the answer according to my current understanding</em></p>
  <blockquote>
  <p>Distributional semantics are <strong>computed from context statistics</strong>.</p>
  </blockquote>
  <p>It is clear from the statement that the embedding of a word, in case of distributional semantics, is computed from the context statistics i.e., based on the contexts in which the word occurs.</p>
  <blockquote>
  <p>Distributed semantics are a <strong>related but distinct</strong> idea: that meaning
  can be represented by numerical vectors rather than <strong>symbolic
  structures.</strong></p>
  </blockquote>
  <p>This means that embeddings obtained from distributed semantics mayn't be obtained from the symbolic structures. Now, we need to understand what is meant by symbolic structure in this case. It can be same as that of context of a word. We can understand it from the following <a href="https://www.sciencedirect.com/topics/computer-science/symbol-structure" rel="nofollow noreferrer">definition of symbol structure</a></p>
  <blockquote>
  <p>A physical symbol system consists of a set of entities, called
  symbols, which are physical patterns that can occur as components of
  another type of entity called an expression (symbol structure). Thus a
  symbol structure is composed of a number of instances (or tokens) of
  symbols related in some physical way (such as one token being next to
  another).</p>
  </blockquote>
  <p>So, it can be understood that distributed semantics are the embeddings obtained <strong>not only through the context statistics as in the case of distributional semantics</strong>. For example, there are distributed representations beyond distributional statistics, in which embeddings are calculated from the internal structure of words and not from the context in which the word occurs (<a href="https://cseweb.ucsd.edu/%7Ennakashole/teaching/eisenstein-nov18.pdf" rel="nofollow noreferrer">p 341</a>). One can understand it from the following excerpt from the same page</p>
  <blockquote>
  <p>How  can  word-internal  structure  be  incorporated  into  word
  representations?   One approach is to construct word representations from embeddings of the characters or morphemes.</p>
  </blockquote>
  <p>Thus, to be concise, the embedding for the word <code>cat</code> is only obtained using the context statistics of <code>cat</code> in case of distributional semantics and in case of distributed semantics, the embedding of the word <code>millicuries</code> can be calculated from the embeddings the morphemes <span class="math-container">$milli,curie,s$</span> rather than the context statistics of the word <code>millicuries</code> since it is a rare word which is unlikely to have reliable context information available.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-11T10:40:00.973Z
Answers.Text: I don't think that's right. Characters don't play a role in here; I think it refers to the context of other _tokens_. In "the cat sat on the mat" the symbolic context would be "the", "sat", "on", "the", and "mat"; in a distributional representation these tokens would be replaced by vectors.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-11T10:48:48.037Z
Answers.Text: @OliverMason please check now, I updated/
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-11T11:14:02.090Z
Answers.Text: looks fine to me now.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-11T10:28:13.657Z
Answers.LastActivityDate: 2021-06-11T11:13:40.700Z
Answers.LastEditDate: 2021-06-11T11:13:40.700Z
Answers.UserDisplayName: hanugm
Body: <p>While studying word embeddings in natural language processing, I encountered the following statement on page 327 of the textbook <a href="https://cseweb.ucsd.edu/%7Ennakashole/teaching/eisenstein-nov18.pdf" rel="nofollow noreferrer">Natural Language Processing by Jacob Eisenstein</a></p>
  <blockquote>
  <p>Distributional semantics are computed from context statistics. Distributed semantics are a related but distinct idea: that meaning can be represented by numerical vectors rather than <strong>symbolic structures</strong>.</p>
  </blockquote>
  <p>The dissimilarity between them is that distributed semantics represent the meaning of a word by a vector of numbers. Distributional semantics represent the meaning of a word by symbolic structure (inferred from paragraph).</p>
  <p>I can say, in distributed semantics, the word cat can be represented by the vector <span class="math-container">$[23, 43,21,16]$</span> (for example).</p>
  <p>Similarly, please, give me a small example of how the meaning of a word is represented by symbolic structure (which should not be necessarily correct).</p>
  <p>What is meant by symbolic structure here?</p>
  
Tags: <comparison><terminology><word-embedding><books>
Title: What is the exact difference between distributional semantics and distributed semantics?
UserDisplayName: hanugm
ThreadText: # What is the exact difference between distributional semantics and distributed semantics?
  
  (Asked by: username_0 on 2021-06-09)
  
  While studying word embeddings in natural language processing, I encountered the following statement on page 327 of the textbook [Natural Language Processing by Jacob Eisenstein](https://cseweb.ucsd.edu/%7Ennakashole/teaching/eisenstein-nov18.pdf)
  
  > Distributional semantics are computed from context statistics. Distributed semantics are a related but distinct idea: that meaning can be represented by numerical vectors rather than **symbolic structures**.
  
  The dissimilarity between them is that distributed semantics represent the meaning of a word by a vector of numbers. Distributional semantics represent the meaning of a word by symbolic structure (inferred from paragraph).
  
  I can say, in distributed semantics, the word cat can be represented by the vector $[23, 43,21,16]$ (for example).
  
  Similarly, please, give me a small example of how the meaning of a word is represented by symbolic structure (which should not be necessarily correct).
  
  What is meant by symbolic structure here?
  
  **Tags:** comparison, terminology, word-embedding, books
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-06-09. Score: 1
  
  I can't really make much sense of Eisenstein's distinction between *distributional* and *distributed*. And I think in your question you actually mix up the two terms as well, as distribut**ed** semantics involve symbolic structures, whereas distribut**ional** semantics are numerical vectors according to his definition. EDIT: actually, he seems to mix it up himself there?! Very unclear paragraph there.
  
  I can only imagine that the symbolic structures he refers to here are semantic networks and the like, as in
  
  > (is-a feline mammal)  
  > (is-a lion feline)  
  > (has-a feline tail)
  
  Here the meaning of *lion*, as a feline mammal with a tail, is defined through a symbolic structure, and not in reference to the context of usage. Why this should be distributed, I can only guess: the meaning components are split over a set of statements, which build up a larger structure perhaps?
  
  It could, of course, be the case that this is covered elsewhere in the book — I haven't had the time to look through all of it.
  
  UPDATE: Thinking more about this, perhaps he means that distributional semantics are representations where each word is a straight co-occurrence vector, ie a vector as large as the words used to define contexts, while distributed semantics is similar, but it's a different vector which is created through processing the contexts (and could thus be smaller)?
  
  
  ---
  
  ### Answer by username_0 on 2021-06-11. Score: 0
  
  *I am writing the answer according to my current understanding*
  
  > Distributional semantics are **computed from context statistics**.
  
  It is clear from the statement that the embedding of a word, in case of distributional semantics, is computed from the context statistics i.e., based on the contexts in which the word occurs.
  
  > Distributed semantics are a **related but distinct** idea: that meaning
  > can be represented by numerical vectors rather than **symbolic
  > structures.**
  
  This means that embeddings obtained from distributed semantics mayn't be obtained from the symbolic structures. Now, we need to understand what is meant by symbolic structure in this case. It can be same as that of context of a word. We can understand it from the following [definition of symbol structure](https://www.sciencedirect.com/topics/computer-science/symbol-structure)
  
  > A physical symbol system consists of a set of entities, called
  > symbols, which are physical patterns that can occur as components of
  > another type of entity called an expression (symbol structure). Thus a
  > symbol structure is composed of a number of instances (or tokens) of
  > symbols related in some physical way (such as one token being next to
  > another).
  
  So, it can be understood that distributed semantics are the embeddings obtained **not only through the context statistics as in the case of distributional semantics**. For example, there are distributed representations beyond distributional statistics, in which embeddings are calculated from the internal structure of words and not from the context in which the word occurs ([p 341](https://cseweb.ucsd.edu/%7Ennakashole/teaching/eisenstein-nov18.pdf)). One can understand it from the following excerpt from the same page
  
  > How can word-internal structure be incorporated into word
  > representations? One approach is to construct word representations from embeddings of the characters or morphemes.
  
  Thus, to be concise, the embedding for the word `cat` is only obtained using the context statistics of `cat` in case of distributional semantics and in case of distributed semantics, the embedding of the word `millicuries` can be calculated from the embeddings the morphemes $milli,curie,s$ rather than the context statistics of the word `millicuries` since it is a rare word which is unlikely to have reliable context information available.
  
  > **Comments:**
  > **username_1**: I don't think that's right. Characters don't play a role in here; I think it refers to the context of other _tokens_. In "the cat sat on the mat" the symbolic context would be "the", "sat", "on", "the", and "mat"; in a distributional representation these tokens would be replaced by vectors.  
  > **username_0**: @OliverMason please check now, I updated/  
  > **username_1**: looks fine to me now.  
  
  ---
  
  