Answers.Body: <p>The specific term you are looking for is &quot;word embedding&quot; and not just &quot;embedding&quot;.</p>
  <h3>How to numerically represent textual data?</h3>
  <p>Neural networks (typically) require as inputs (and produce as outputs) numerical data (i.e. numbers, vectors, matrices, or higher-dimensional arrays). So, when processing textual data, we first need to encode (or convert) the text into a numerical representation. There are different ways to do it, such as</p>
  <ul>
  <li><p><strong>one-hot encoding</strong> (in that case, if you have 70000 words, you would have <em>sparse</em> vectors with 70000 entries where only one of those entries is equal to <span class="math-container">$1$</span> and all other entries are <span class="math-container">$0$</span>: see <a href="https://www.tensorflow.org/tutorials/text/word_embeddings" rel="nofollow noreferrer">this article </a> for more info)</p>
  </li>
  <li><p><strong>map each word to a number</strong> (in this case, you would have 70000 numbers, one for each word)</p>
  </li>
  <li><p><strong>word embeddings</strong></p>
  </li>
  </ul>
  <p>Each of these representations has different benefits and drawbacks. For instance, if you map each word to a number, then you just need to keep track of <span class="math-container">$70000$</span> numbers. In the case of one-hot encoding or word embeddings, you will need more memory. However, nowadays, word embeddings are widely used in natural language processing/understanding/generation tasks (and given that your question is about word embeddings), so let me briefly describe them.</p>
  <h3>Word embeddings</h3>
  <p>There are different word embedding techniques (such as <a href="https://www.tensorflow.org/tutorials/text/word2vec" rel="nofollow noreferrer">word2vec</a>). However, they are all based on the same ideas</p>
  <ol>
  <li><p>Words that are similar (or related) in meaning should be mapped to vectors (i.e. the &quot;word embeddings&quot;) that are also similar in some sense (for instance, their cosine similarity should be high). For instance, the words &quot;man&quot; and &quot;boy&quot; should be mapped to vectors that are similar.</p>
  </li>
  <li><p>These word embeddings are learned (rather than hard-coded or manually specified) given the data</p>
  </li>
  <li><p>The size of the word embeddings is a hyper-parameter (this should answer your question!)</p>
  </li>
  </ol>
  <h3>Hyper-parameters</h3>
  <p>To answer your question(s) more directly, the choice of the dimension of the embeddings or the number of &quot;hidden features&quot; (which are both <em>hyper-parameters</em>) was probably more or less arbitrary or based on the instructor's experience. In general, it is difficult to determine the optimal choice of any hyper-parameter. Sometimes you can just use numbers that other people have used in the past and have noticed that work &quot;well enough&quot;. If you really want to find more appropriate values of the hyper-parameters, you could use some hyper-parameter optimization technique, such as <em>Bayesian optimization</em> or a simple <em>grid search</em>.</p>
  <h3>Further reading</h3>
  <p>You can find many resources online that explain the concept of &quot;word embeddings&quot; more in detail. For instance</p>
  <ul>
  <li><a href="https://machinelearningmastery.com/what-are-word-embeddings/" rel="nofollow noreferrer">What Are Word Embeddings for Text?</a> (by Jason Brownlee)</li>
  <li><a href="https://www.tensorflow.org/tutorials/text/word_embeddings" rel="nofollow noreferrer">Word embeddings</a> (A TensorFlow article/tutorial on word embeddings).</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T04:25:16.383Z
Answers.Text: thanks! the references mentioned in your answer is hugely informative and helpful!
Answers.UserDisplayName: Franva
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T14:43:45.763Z
Answers.Text: It's useful to know, if the instructor hasn't mentioned it, that you develop intuitions based on empirical (experimental) evidence in this space.  In other words, why 256?  Because I've done this before and 256 seemed to work well. :). It can be more specific... If you have a high degree of expertise in a domain, or you apply statistical methods to identify relevant features, you can select features and other parameters with more rigorous methods, but otherwise, it's guessing with fancy names. :)
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T14:47:38.357Z
Answers.Text: @DavidHoelzer You're right that those numbers might have been picked by guessing. My point is that these are hyper-parameters, so there isn't a strict rule to decide their values. You can use some hyper-parameter optimization technique, but more often than not you will use some specific value because someone else used it (and that's good enough for your purposes for now). Again, of course, this is not a very rigorous/scientific way of solving the problem and, if (small) differences in performance really count, you should solve this hyper-parameter optimization problem more rigorously.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T19:44:37.220Z
Answers.Text: Yes, exactly... I wasn’t disagreeing at all... Just saying the same thing a little differently and in a way I find my students understand it better.. :)
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-06T12:50:09.397Z
Answers.LastActivityDate: 2020-11-06T13:01:06.127Z
Answers.LastEditDate: 2020-11-06T13:01:06.127Z
Answers.UserDisplayName: nbro
Answers.Body: <p>I finally grasped the concept of word embedding. Thanks to @nbro, after reading the 2 articles s/he recommended</p>
  <ol>
  <li><p><a href="https://machinelearningmastery.com/what-are-word-embeddings/" rel="nofollow noreferrer">What Are Word Embeddings for Text?</a> and</p>
  </li>
  <li><p><a href="https://www.tensorflow.org/tutorials/text/word_embeddings" rel="nofollow noreferrer">Word embeddings</a></p>
  </li>
  </ol>
  <p>the 1st article gives me a good idea about the big picture of the Word Embeddings; whereas the 2nd article is actually the one which clears my mind.</p>
  <p>I am an visual person, I understand things better if I could see how the things, in this case the Word Embeddings, look like(This answered my 2nd questions).</p>
  <p><a href="https://i.sstatic.net/XENZd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XENZd.png" alt="Word Embeddings" /></a></p>
  <p>After seeing this image, my 1st question is answered and I realized that Word Embeddings is a 2 dimensional array where the number of rows of the array is decided by the number of unique words in your vocabulary and the columns/width is decided by yourself. Normally between 8 up to 1024 according to the 2nd article.</p>
  <p>The columns/width within the course I am learning from is called embedding_dim, which I found hard to comprehend. Since each word embedding is a vector (this answered my 3rd question), for example the cat is [1.2, -0.1, 4.3, 3.2], and the vector is a meta concept for me which is easy to understand, I would like to call the embedding_dim : <em><strong>embedding_vector_width</strong></em> or <em><strong>embedding_vector_length</strong></em>.</p>
  <p>For the 256 hidden features, how many of them would be enough, I think it's the same concept of how to figure out how many embedding_vector_width should be.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T13:03:36.163Z
Answers.LastActivityDate: 2020-11-07T13:03:36.163Z
Answers.UserDisplayName: Franva
Body: <p>I am learning PyTorch on Udacity. In lesson 8, section 11: <a href="https://classroom.udacity.com/courses/ud188/lessons/2f4910ee-6d67-47df-97e2-f35db67cbc19/concepts/062bfbc6-34c8-4e5c-b072-6479eca5a385" rel="nofollow noreferrer">Training the Model</a>, the instructor writes:</p>
  <blockquote>
  <p>Then I have my embedding and hidden dimension. The embedding dimension is just a smaller representation of my vocabulary of 70k words and I think any value between like 200 and 500 or so would work, here. I've chosen 400. Similarly, for our hidden dimension, I think 256 hidden features should be enough to distinguish between positive and negative reviews.</p>
  </blockquote>
  <p>There are more than 70000 different words. How could those more than 70000 unique words be represented by just 400 embeddings? How does an embedding look like? Is it a number?</p>
  <p>Moreover, why would 256 hidden features be enough?</p>
  
Tags: <recurrent-neural-networks><hyperparameter-optimization><word-embedding><hyper-parameters><sentiment-analysis>
Title: Why is an embedding of dimension 400 enough to represent 70000 words?
UserDisplayName: Franva
ThreadText: # Why is an embedding of dimension 400 enough to represent 70000 words?
  
  (Asked by: username_0 on 2020-11-06)
  
  I am learning PyTorch on Udacity. In lesson 8, section 11: [Training the Model](https://classroom.udacity.com/courses/ud188/lessons/2f4910ee-6d67-47df-97e2-f35db67cbc19/concepts/062bfbc6-34c8-4e5c-b072-6479eca5a385), the instructor writes:
  
  > Then I have my embedding and hidden dimension. The embedding dimension is just a smaller representation of my vocabulary of 70k words and I think any value between like 200 and 500 or so would work, here. I've chosen 400. Similarly, for our hidden dimension, I think 256 hidden features should be enough to distinguish between positive and negative reviews.
  
  There are more than 70000 different words. How could those more than 70000 unique words be represented by just 400 embeddings? How does an embedding look like? Is it a number?
  
  Moreover, why would 256 hidden features be enough?
  
  **Tags:** recurrent-neural-networks, hyperparameter-optimization, word-embedding, hyper-parameters, sentiment-analysis
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-11-06. Score: 1 | Accepted answer
  
  The specific term you are looking for is "word embedding" and not just "embedding".
  
  ### How to numerically represent textual data?
  
  Neural networks (typically) require as inputs (and produce as outputs) numerical data (i.e. numbers, vectors, matrices, or higher-dimensional arrays). So, when processing textual data, we first need to encode (or convert) the text into a numerical representation. There are different ways to do it, such as
  
  * **one-hot encoding** (in that case, if you have 70000 words, you would have *sparse* vectors with 70000 entries where only one of those entries is equal to $1$ and all other entries are $0$: see [this article](https://www.tensorflow.org/tutorials/text/word_embeddings)  for more info)
  * **map each word to a number** (in this case, you would have 70000 numbers, one for each word)
  * **word embeddings**
  
  Each of these representations has different benefits and drawbacks. For instance, if you map each word to a number, then you just need to keep track of $70000$ numbers. In the case of one-hot encoding or word embeddings, you will need more memory. However, nowadays, word embeddings are widely used in natural language processing/understanding/generation tasks (and given that your question is about word embeddings), so let me briefly describe them.
  
  ### Word embeddings
  
  There are different word embedding techniques (such as [word2vec](https://www.tensorflow.org/tutorials/text/word2vec)). However, they are all based on the same ideas
  
  1. Words that are similar (or related) in meaning should be mapped to vectors (i.e. the "word embeddings") that are also similar in some sense (for instance, their cosine similarity should be high). For instance, the words "man" and "boy" should be mapped to vectors that are similar.
  2. These word embeddings are learned (rather than hard-coded or manually specified) given the data
  3. The size of the word embeddings is a hyper-parameter (this should answer your question!)
  
  ### Hyper-parameters
  
  To answer your question(s) more directly, the choice of the dimension of the embeddings or the number of "hidden features" (which are both *hyper-parameters*) was probably more or less arbitrary or based on the instructor's experience. In general, it is difficult to determine the optimal choice of any hyper-parameter. Sometimes you can just use numbers that other people have used in the past and have noticed that work "well enough". If you really want to find more appropriate values of the hyper-parameters, you could use some hyper-parameter optimization technique, such as *Bayesian optimization* or a simple *grid search*.
  
  ### Further reading
  
  You can find many resources online that explain the concept of "word embeddings" more in detail. For instance
  
  * [What Are Word Embeddings for Text?](https://machinelearningmastery.com/what-are-word-embeddings/) (by Jason Brownlee)
  * [Word embeddings](https://www.tensorflow.org/tutorials/text/word_embeddings) (A TensorFlow article/tutorial on word embeddings).
  
  > **Comments:**
  > **username_0**: thanks! the references mentioned in your answer is hugely informative and helpful!  
  > **username_2**: It's useful to know, if the instructor hasn't mentioned it, that you develop intuitions based on empirical (experimental) evidence in this space.  In other words, why 256?  Because I've done this before and 256 seemed to work well. :). It can be more specific... If you have a high degree of expertise in a domain, or you apply statistical methods to identify relevant features, you can select features and other parameters with more rigorous methods, but otherwise, it's guessing with fancy names. :)  
  > **username_1**: @DavidHoelzer You're right that those numbers might have been picked by guessing. My point is that these are hyper-parameters, so there isn't a strict rule to decide their values. You can use some hyper-parameter optimization technique, but more often than not you will use some specific value because someone else used it (and that's good enough for your purposes for now). Again, of course, this is not a very rigorous/scientific way of solving the problem and, if (small) differences in performance really count, you should solve this hyper-parameter optimization problem more rigorously.  
  > **username_2**: Yes, exactly... I wasn’t disagreeing at all... Just saying the same thing a little differently and in a way I find my students understand it better.. :)  
  
  ---
  
  ### Answer by username_0 on 2020-11-07. Score: 1
  
  I finally grasped the concept of word embedding. Thanks to @nbro, after reading the 2 articles s/he recommended
  
  1. [What Are Word Embeddings for Text?](https://machinelearningmastery.com/what-are-word-embeddings/) and
  2. [Word embeddings](https://www.tensorflow.org/tutorials/text/word_embeddings)
  
  the 1st article gives me a good idea about the big picture of the Word Embeddings; whereas the 2nd article is actually the one which clears my mind.
  
  I am an visual person, I understand things better if I could see how the things, in this case the Word Embeddings, look like(This answered my 2nd questions).
  
  [![Word Embeddings](https://i.sstatic.net/XENZd.png)](https://i.sstatic.net/XENZd.png)
  
  After seeing this image, my 1st question is answered and I realized that Word Embeddings is a 2 dimensional array where the number of rows of the array is decided by the number of unique words in your vocabulary and the columns/width is decided by yourself. Normally between 8 up to 1024 according to the 2nd article.
  
  The columns/width within the course I am learning from is called embedding\_dim, which I found hard to comprehend. Since each word embedding is a vector (this answered my 3rd question), for example the cat is [1.2, -0.1, 4.3, 3.2], and the vector is a meta concept for me which is easy to understand, I would like to call the embedding\_dim : ***embedding\_vector\_width*** or ***embedding\_vector\_length***.
  
  For the 256 hidden features, how many of them would be enough, I think it's the same concept of how to figure out how many embedding\_vector\_width should be.
  
  
  ---
  
  