Answers.Body: <p>The information you are probably missing is that word embeddings are learned on the basis of context. For example, you might try to predict a vector for a word from the wordvectors of the other words in the same sentence. </p>
  
  <p>This way word vectors of words that occur in similar contexts will turn out to be similar. You can think of it as word vectors not encoding the word themselves but the contexts in which they are used. Of course ultimately that is the same. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-02T09:09:59.297Z
Answers.LastActivityDate: 2019-10-02T09:09:59.297Z
Answers.UserDisplayName: BlindKungFuMaster
Answers.Body: <p>The <strong>subword-based embedding</strong> is rather visual and easily understandable. However, the <strong>autoencoder embedding</strong> is what machines understand the componential meaning of words.</p>
  
  <p>1) An <strong>autoencoder embedding layer</strong> can be trained together with other layers to fit with the relation of data in dataset.</p>
  
  <p>2) Or the <strong>embedding layer can be kept unchanged</strong> as used as a function, <strong>required</strong> that the embedding layer must have been trained on a similar task like said in 1)</p>
  
  <p>And as stated in the question, embedding is important in NLP as <strong>word indices don't show the full meanings of words</strong>, they must be <strong>separated</strong> into embedding values <strong>for better efficiency</strong>.</p>
  
  <p>TensorFlow random uniform trainable embedding: <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding" rel="nofollow noreferrer">https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding</a></p>
  
  <p>TensorFlow utility class for creating subword-based encoder: <a href="https://www.tensorflow.org/datasets/api_docs/python/tfds/features/text/SubwordTextEncoder" rel="nofollow noreferrer">https://www.tensorflow.org/datasets/api_docs/python/tfds/features/text/SubwordTextEncoder</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-01T09:15:33.270Z
Answers.LastActivityDate: 2019-10-02T05:23:29.520Z
Answers.LastEditDate: 2019-10-02T05:23:29.520Z
Answers.UserDisplayName: Dan D.
Body: <p>People say <strong>embedding</strong> is necessary in NLP because if using just the word indices, the <strong>efficiency is not high</strong> as <strong>similar words are supposed to be related to each other</strong>. However, I still don't truly get it why.</p>
  
  <p>The <strong>subword-based embedding</strong> (aka syllable-based embedding) is understandable, for example:</p>
  
  <pre><code>biology   --&gt; bio-lo-gy
  biologist --&gt; bio-lo-gist
  </code></pre>
  
  <p>For the 2 words above, when turning them into syllable-based embeddings, it's good because the 2 words will be related to each other due to the sharing syllables: <code>bio</code>, and <code>lo</code>.</p>
  
  <p>However, it's hard to understand the <code>autoencoder</code>, it turns an index value into vector, then feed these vectors to DNN. Autoencoder can turn vectors back to words too.</p>
  
  <p><strong>How does autoencoder make words related to each other?</strong></p>
  
Tags: <deep-learning><natural-language-processing><tensorflow><word-embedding><hidden-layers>
Title: Why is embedding important in NLP, and how does autoencoder work?
UserDisplayName: Dan D.
ThreadText: # Why is embedding important in NLP, and how does autoencoder work?
  
  (Asked by: username_0 on 2019-10-01)
  
  People say **embedding** is necessary in NLP because if using just the word indices, the **efficiency is not high** as **similar words are supposed to be related to each other**. However, I still don't truly get it why.
  
  The **subword-based embedding** (aka syllable-based embedding) is understandable, for example:
  
  ```
  biology   --> bio-lo-gy
  biologist --> bio-lo-gist
  
  ```
  
  For the 2 words above, when turning them into syllable-based embeddings, it's good because the 2 words will be related to each other due to the sharing syllables: `bio`, and `lo`.
  
  However, it's hard to understand the `autoencoder`, it turns an index value into vector, then feed these vectors to DNN. Autoencoder can turn vectors back to words too.
  
  **How does autoencoder make words related to each other?**
  
  **Tags:** deep-learning, natural-language-processing, tensorflow, word-embedding, hidden-layers
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-10-02. Score: 1 | Accepted answer
  
  The information you are probably missing is that word embeddings are learned on the basis of context. For example, you might try to predict a vector for a word from the wordvectors of the other words in the same sentence.
  
  This way word vectors of words that occur in similar contexts will turn out to be similar. You can think of it as word vectors not encoding the word themselves but the contexts in which they are used. Of course ultimately that is the same.
  
  
  ---
  
  ### Answer by username_0 on 2019-10-01. Score: 1
  
  The **subword-based embedding** is rather visual and easily understandable. However, the **autoencoder embedding** is what machines understand the componential meaning of words.
  
  1) An **autoencoder embedding layer** can be trained together with other layers to fit with the relation of data in dataset.
  
  2) Or the **embedding layer can be kept unchanged** as used as a function, **required** that the embedding layer must have been trained on a similar task like said in 1)
  
  And as stated in the question, embedding is important in NLP as **word indices don't show the full meanings of words**, they must be **separated** into embedding values **for better efficiency**.
  
  TensorFlow random uniform trainable embedding: <https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding>
  
  TensorFlow utility class for creating subword-based encoder: <https://www.tensorflow.org/datasets/api_docs/python/tfds/features/text/SubwordTextEncoder>
  
  
  ---
  
  