Answers.Body: <p>A <strong>language model</strong> aims to estimate the probability of one or more words given the surrounding words. Given a sentence composed of <span class="math-container">$w_{1},...,w_{i-1},\_ , w_{i+1},..,w_{n}$</span>, you can find which is the i-th missing word using a language model. In this way, you can estimate which is the most probable word using for example the conditional probability <span class="math-container">$P(w_i=w|w_1,…,w_n)$</span>.
  An example of a simple language model is an n-gram where instead of conditioning on all previous words, you look only to the previous n words.</p>
  <p><strong>Word embeddings</strong> are a distributed representation of a word. Instead of using an index or a one-hot encoding to represent a word, a dense vector is used.  If two words have similar embeddings then these words share some properties. These properties are driven by the way embeddings are constructed, for example in word2vec two words with similar embeddings are two words that often appear in the same context, which is not to say they have the same meaning. Sometimes words with opposite meanings can have similar embeddings just because they are placed within the same sentences/contexts.</p>
  <p>The <strong>BLEU score</strong> is a way to quantify the translation quality of an automatic translation. The score aims to look at how different model translation is to human translation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-10T22:45:14.423Z
Answers.LastActivityDate: 2021-06-03T23:16:47.837Z
Answers.LastEditDate: 2021-06-03T23:16:47.837Z
Answers.UserDisplayName: SMattia
Answers.Body: <p><strong>Simplified:</strong> Word Embeddings does not consider context, Language Models does.</p>
  <p>For e.g Word2Vec, GloVe, or fastText, there exists one fixed vector per word.</p>
  <p>Think of the following two sentences:</p>
  <blockquote>
  <p>The fish ate the cat.</p>
  </blockquote>
  <p>and</p>
  <blockquote>
  <p>The cat ate the fish.</p>
  </blockquote>
  <p>If you averaged their word embeddings, they would have the same vector, but, in reality, their meaning (semantic) is very different.</p>
  <p>Then the concept of contextualized word embeddings arose with language models that <em>do</em> consider the context, and give different embeddings depending on the context.</p>
  <p>Both word embeddings (e.g Word2Vec) and language models (e.g BERT) are ways of representing text, where language models capture more information and are considered state-of-the-art for representing natural language in a vectorized format.</p>
  <p><strong>BLEU score</strong> is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Which is not directly related to the difference between traditional word embeddings and contextualized word embeddings (aka language models).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-10T10:45:45.860Z
Answers.LastActivityDate: 2021-03-10T15:45:52.613Z
Answers.LastEditDate: 2021-03-10T15:45:52.613Z
Answers.UserDisplayName: Isbister
Body: <p>I am self-studying applications of deep learning on the NLP and machine translation.</p>
  <p>I am confused about the concepts of &quot;Language Model&quot;, &quot;Word Embedding&quot;, &quot;BLEU Score&quot;.</p>
  <p>It appears to me that a language model is a way to predict the next word given its previous word. Word2vec is the similarity between two tokens. BLEU score is a way to measure the effectiveness of the language model.</p>
  <p>Is my understanding correct? If not, can someone please point me to the right articles, paper, or any other online resources?</p>
  
Tags: <natural-language-processing><comparison><word-embedding><language-model><bleu>
Title: What is the difference between a language model and a word embedding?
UserDisplayName: Exploring
ThreadText: # What is the difference between a language model and a word embedding?
  
  (Asked by: username_0 on 2021-03-09)
  
  I am self-studying applications of deep learning on the NLP and machine translation.
  
  I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score".
  
  It appears to me that a language model is a way to predict the next word given its previous word. Word2vec is the similarity between two tokens. BLEU score is a way to measure the effectiveness of the language model.
  
  Is my understanding correct? If not, can someone please point me to the right articles, paper, or any other online resources?
  
  **Tags:** natural-language-processing, comparison, word-embedding, language-model, bleu
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-03-10. Score: 7 | Accepted answer
  
  **Simplified:** Word Embeddings does not consider context, Language Models does.
  
  For e.g Word2Vec, GloVe, or fastText, there exists one fixed vector per word.
  
  Think of the following two sentences:
  
  > The fish ate the cat.
  
  and
  
  > The cat ate the fish.
  
  If you averaged their word embeddings, they would have the same vector, but, in reality, their meaning (semantic) is very different.
  
  Then the concept of contextualized word embeddings arose with language models that *do* consider the context, and give different embeddings depending on the context.
  
  Both word embeddings (e.g Word2Vec) and language models (e.g BERT) are ways of representing text, where language models capture more information and are considered state-of-the-art for representing natural language in a vectorized format.
  
  **BLEU score** is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Which is not directly related to the difference between traditional word embeddings and contextualized word embeddings (aka language models).
  
  
  ---
  
  ### Answer by username_1 on 2021-03-10. Score: 3
  
  A **language model** aims to estimate the probability of one or more words given the surrounding words. Given a sentence composed of $w\_{1},...,w\_{i-1},\\_ , w\_{i+1},..,w\_{n}$, you can find which is the i-th missing word using a language model. In this way, you can estimate which is the most probable word using for example the conditional probability $P(w\_i=w|w\_1,…,w\_n)$.
  An example of a simple language model is an n-gram where instead of conditioning on all previous words, you look only to the previous n words.
  
  **Word embeddings** are a distributed representation of a word. Instead of using an index or a one-hot encoding to represent a word, a dense vector is used. If two words have similar embeddings then these words share some properties. These properties are driven by the way embeddings are constructed, for example in word2vec two words with similar embeddings are two words that often appear in the same context, which is not to say they have the same meaning. Sometimes words with opposite meanings can have similar embeddings just because they are placed within the same sentences/contexts.
  
  The **BLEU score** is a way to quantify the translation quality of an automatic translation. The score aims to look at how different model translation is to human translation.
  
  
  ---
  
  