Answers.Body: <p>A bag-of-words-model (BOW) is usually used to represent a text: you throw all the words together (as if in a bag), without keeping track of their sequence. This is a gross simplification over a text, as word sequencing plays an important role in creating the meaning of a text. But on the positive side it's easier to handle, eg in information retrieval tasks, where you might not need the precise meaning anyway.</p>
  <p>So the BOW is <em>discrete</em> and <em>symbolic</em>, as it represents each of its elements by a set of words that are contained in it. Nothing numeric in there. You'd calculate the similarity of two items by comparing the two sets, how big is their intersection, and the difference between the two.</p>
  <p>A CBOW is a slight modification: instead of the words, we use vector representations of them; and instead of having <span class="math-container">$n$</span> vectors for the <span class="math-container">$n$</span> surrounding words, they're all added up (formula 14.14) It's still a BOW, as the set of words used to represent an element is now the set of words surrounding it within a certain distance (<span class="math-container">$h$</span>). What makes it <em>continuous</em> is the switch from a set of words (ie symbols) to a vector.</p>
  <p>He contrasts this with a recurrent neural network, where words are represented by a state vector which gets updated after every new word, going back to the very beginning of the text. This would give different representations for the same word occurring in the same localised context, whereas the CBOW would return the same representation.</p>
  <p>For example, for <span class="math-container">$h$</span> being 1 (to keep it simple):</p>
  <blockquote>
  <p>when a word has a meaning, then a word has a purpose.</p>
  </blockquote>
  <p>Now imagine we're interested in the encoding of <em>word</em>: in the recurrent case the first one is <code>when + a + word</code>, whereas the second one is <code>when + a + word + has + a + meaning + , + then + a +word</code> — the sequences here represent the updated state of the network after the respective words have been added.</p>
  <p>In the CBOW case, both occurrences of <em>word</em> are encoded by <code>a + word + has</code> (the word plus/minus one word either side, as <span class="math-container">$h$</span> is 1). So they will be identical.</p>
  <p>To answer your question, <em>continuous</em> here is in contrast to <em>discrete</em> or <em>symbolic</em>, and indeed refers to a numerical vector.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-10T16:58:13.497Z
Answers.LastActivityDate: 2021-06-10T16:58:13.497Z
Answers.UserDisplayName: Oliver Mason
Body: <p>The word continuous in mathematics is a property of either a set or a function that says that the underlying object has no discontinuity in the range mentioned. If the object is a set, then <span class="math-container">$[-1,1]$</span> is a continuous one while <span class="math-container">$\{-1, +1\}$</span> is not. Similarly, a function is said to be continuous if the actual value and the limiting value at every point in the domain are equal.</p>
  <p>Now, coming to CBOW. I read the following statement from p:334 of <a href="https://cseweb.ucsd.edu/%7Ennakashole/teaching/eisenstein-nov18.pdf" rel="nofollow noreferrer">Natural Language Processing by Jacob Eisenstein</a></p>
  <blockquote>
  <p>Thus, CBOW is a bag-of-words model, because the order of the context words does not matter; <strong>it is continuous, because rather than conditioning on the words themselves, we condition on a <em>continuous vector</em> constructed from the word embeddings</strong>.</p>
  </blockquote>
  <p>What is meant by <em>continuous</em> in this case?  Does <em>continuous vector</em> stand for a vector of real numbers?</p>
  
Tags: <terminology><books><bag-of-words><cbow>
Title: What is the meaning of "continuous" in a continuous bag-of-words model?
UserDisplayName: hanugm
ThreadText: # What is the meaning of "continuous" in a continuous bag-of-words model?
  
  (Asked by: username_0 on 2021-06-10)
  
  The word continuous in mathematics is a property of either a set or a function that says that the underlying object has no discontinuity in the range mentioned. If the object is a set, then $[-1,1]$ is a continuous one while $\{-1, +1\}$ is not. Similarly, a function is said to be continuous if the actual value and the limiting value at every point in the domain are equal.
  
  Now, coming to CBOW. I read the following statement from p:334 of [Natural Language Processing by Jacob Eisenstein](https://cseweb.ucsd.edu/%7Ennakashole/teaching/eisenstein-nov18.pdf)
  
  > Thus, CBOW is a bag-of-words model, because the order of the context words does not matter; **it is continuous, because rather than conditioning on the words themselves, we condition on a *continuous vector* constructed from the word embeddings**.
  
  What is meant by *continuous* in this case? Does *continuous vector* stand for a vector of real numbers?
  
  **Tags:** terminology, books, bag-of-words, cbow
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-10. Score: 1
  
  A bag-of-words-model (BOW) is usually used to represent a text: you throw all the words together (as if in a bag), without keeping track of their sequence. This is a gross simplification over a text, as word sequencing plays an important role in creating the meaning of a text. But on the positive side it's easier to handle, eg in information retrieval tasks, where you might not need the precise meaning anyway.
  
  So the BOW is *discrete* and *symbolic*, as it represents each of its elements by a set of words that are contained in it. Nothing numeric in there. You'd calculate the similarity of two items by comparing the two sets, how big is their intersection, and the difference between the two.
  
  A CBOW is a slight modification: instead of the words, we use vector representations of them; and instead of having $n$ vectors for the $n$ surrounding words, they're all added up (formula 14.14) It's still a BOW, as the set of words used to represent an element is now the set of words surrounding it within a certain distance ($h$). What makes it *continuous* is the switch from a set of words (ie symbols) to a vector.
  
  He contrasts this with a recurrent neural network, where words are represented by a state vector which gets updated after every new word, going back to the very beginning of the text. This would give different representations for the same word occurring in the same localised context, whereas the CBOW would return the same representation.
  
  For example, for $h$ being 1 (to keep it simple):
  
  > when a word has a meaning, then a word has a purpose.
  
  Now imagine we're interested in the encoding of *word*: in the recurrent case the first one is `when + a + word`, whereas the second one is `when + a + word + has + a + meaning + , + then + a +word` — the sequences here represent the updated state of the network after the respective words have been added.
  
  In the CBOW case, both occurrences of *word* are encoded by `a + word + has` (the word plus/minus one word either side, as $h$ is 1). So they will be identical.
  
  To answer your question, *continuous* here is in contrast to *discrete* or *symbolic*, and indeed refers to a numerical vector.
  
  
  ---
  
  