Answers.Body: <p>An embedding is a vector that <strong>semantically</strong> represents an object/input, which, in the context of NLP, can be a character, word, or sentence, depending on the task. The main property of embeddings is that they are close/similar to each other (for some notion of similarity, e.g. the cosine similarity) if the corresponding original objects/inputs also have similar meanings or are <em>contextually</em> related. So, for example, the words &quot;boy&quot; and &quot;man&quot; are expected to be mapped to vectors/embeddings that are close to each other. There are different ways to create these embeddings. An example is <a href="https://arxiv.org/pdf/1301.3781.pdf" rel="nofollow noreferrer">word2vec</a>. Note that the notion of an embedding also applies to other contexts, e.g. geometric deep learning. You can read more about this topic in <a href="https://web.stanford.edu/%7Ejurafsky/slp3/ed3book.pdf#page=104" rel="nofollow noreferrer">chapter 6</a> of <a href="https://web.stanford.edu/%7Ejurafsky/slp3/ed3book.pdf" rel="nofollow noreferrer">this book</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-30T21:21:39.397Z
Answers.Text: Ahh, yes, I'm in the early chapters of the Jurafsky and Martin textbook. Thanks for the clarification.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-31T09:08:59.617Z
Answers.Text: It's worth adding where the similarity comes from: usually embeddings are based on the surrounding context of a word, and both 'man' and 'boy' tend to be used in similar environments, at least more than 'man' and 'quickly'. The more the usage of two words overlaps, the more similar they tend to be.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-30T21:03:11.560Z
Answers.LastActivityDate: 2021-05-30T21:03:11.560Z
Answers.UserDisplayName: nbro
Body: <p>When reading about NLP, I saw it said that &quot;input embeddings&quot; are a main element of encoder-decoder learning frameworks for sequence modelling. What is an &quot;input embedding&quot; in the context of NLP?</p>
  
Tags: <natural-language-processing><embeddings>
Title: What is an "input embedding" in the context of NLP?
UserDisplayName: The Pointer
ThreadText: # What is an "input embedding" in the context of NLP?
  
  (Asked by: username_0 on 2021-05-30)
  
  When reading about NLP, I saw it said that "input embeddings" are a main element of encoder-decoder learning frameworks for sequence modelling. What is an "input embedding" in the context of NLP?
  
  **Tags:** natural-language-processing, embeddings
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-30. Score: 1 | Accepted answer
  
  An embedding is a vector that **semantically** represents an object/input, which, in the context of NLP, can be a character, word, or sentence, depending on the task. The main property of embeddings is that they are close/similar to each other (for some notion of similarity, e.g. the cosine similarity) if the corresponding original objects/inputs also have similar meanings or are *contextually* related. So, for example, the words "boy" and "man" are expected to be mapped to vectors/embeddings that are close to each other. There are different ways to create these embeddings. An example is [word2vec](https://arxiv.org/pdf/1301.3781.pdf). Note that the notion of an embedding also applies to other contexts, e.g. geometric deep learning. You can read more about this topic in [chapter 6](https://web.stanford.edu/%7Ejurafsky/slp3/ed3book.pdf#page=104) of [this book](https://web.stanford.edu/%7Ejurafsky/slp3/ed3book.pdf).
  
  > **Comments:**
  > **username_0**: Ahh, yes, I'm in the early chapters of the Jurafsky and Martin textbook. Thanks for the clarification.  
  > **username_2**: It's worth adding where the similarity comes from: usually embeddings are based on the surrounding context of a word, and both 'man' and 'boy' tend to be used in similar environments, at least more than 'man' and 'quickly'. The more the usage of two words overlaps, the more similar they tend to be.  
  
  ---
  
  