Answers.Body: <p>My understanding is that the ASCII encoding would not get the best performance or results from the RNN because the ASCII codes for each character are not meaningful; they are arbitrary. If the number of each ASCII code represented something meaningful about the letter, it would work better. But they don't.</p>
  <p>The same principles apply as when deciding how to encode any categorical data. If your categories are ordinal (eg. 'First', 'Second' .. or 'Age group 18-24', Age group 25-35' .. or even 'Social Class E', 'Social Class D' ..), then assigning a single numerical value to each class might work well. But in categorical data where there is no meaningful order, one hot encoding will work better.</p>
  <p>This is an example of the principle of giving neural networks the most <strong>expressive</strong> data that we can. In the case of non-ordinal, arbitrary categories, one-hot is more expressive to the next layer of neurons (will stimulate them more distinctly) than using a numerical encoding.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-16T11:21:15.597Z
Answers.LastActivityDate: 2021-09-16T11:21:15.597Z
Answers.UserDisplayName: Jamie McLaughlin
Answers.Body: <p>The request for input encoding is that mathematical difference between encoded inputs (numerical subtraction in absolute value of vector norm, etc) must be proportional to logical (real world) dis-similitude between the objects they represent.</p>
  <p>Lets say by example the 3 first uppercase letters. In one-hot encoding they will be:</p>
  <pre><code>'A' : (1,0,0)
  'B' : (0,1,0)
  'C' : (0,0,1)
  </code></pre>
  <p>Note that the distance between any two pairs is 0 if the letters are the same and 2 if they differ.</p>
  <p>Compare with ASCII in binary as encoding:</p>
  <pre><code>'A' : 0100 0001  
  'B' : 0100 0010  
  'C' : 0100 0011
  </code></pre>
  <p>Note distance between 'A' and 'B' is 2, but distance between 'A' and 'C' is 1. Thus, we are saying that an 'A' is more similar to a 'C' than a 'B'. Something artificial that will cause network errors.</p>
  <p>Equivalent problem appears when using ASCII as integer:</p>
  <pre><code>'A' : 65
  'B' : 66  
  'C' : 67
  </code></pre>
  <p>we are saying that distance between 'A' and 'C' is 2, twice the distance between 'A' and 'B'. Again, something that will disturb network performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T16:14:53.987Z
Answers.LastActivityDate: 2022-02-13T16:20:20.977Z
Answers.LastEditDate: 2022-02-13T16:20:20.977Z
Answers.UserDisplayName: pasaba por aqui
Body: <p>I've mostly seen (e.g. in  <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow noreferrer">The Unreasonable Effectiveness of Recurrent Neural Networks</a>) that when training RNN on text for something like language modeling, the text is usually featurized character-by-character using a 1-hot encoding.</p>
  
  <p>For example, the text "hello" would be represented like</p>
  
  <pre><code>{h: 1, e: 0, l: 0, o: 0}
  {h: 0, e: 1, l: 0, o: 0}
  {h: 0, e: 0, l: 1, o: 0}
  {h: 0, e: 0, l: 1, o: 0}
  {h: 0, e: 0, l: 0, o: 1}
  </code></pre>
  
  <p>I was wondering if one could just as well use the ASCII encoding of the text and feed the bits in one by one. So the input "hello" would be input like</p>
  
  <pre><code>0110100001100101011011000110110001101111 
  </code></pre>
  
  <p>Would the RNN have a disproportionately harder time having to figure out how the arbitrary and complex 8-bit ASCII encoding should be used? Or would the ASCII encoding lead to about the same performance as the nicer 1-hot encoding?</p>
  
Tags: <neural-networks><machine-learning><natural-language-processing><recurrent-neural-networks><long-short-term-memory>
Title: Training RNN's on text: Can you use an ASCII encoding just as well as a one-hot character encoding?
UserDisplayName: chausies
ThreadText: # Training RNN's on text: Can you use an ASCII encoding just as well as a one-hot character encoding?
  
  (Asked by: username_0 on 2017-11-05)
  
  I've mostly seen (e.g. in [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)) that when training RNN on text for something like language modeling, the text is usually featurized character-by-character using a 1-hot encoding.
  
  For example, the text "hello" would be represented like
  
  ```
  {h: 1, e: 0, l: 0, o: 0}
  {h: 0, e: 1, l: 0, o: 0}
  {h: 0, e: 0, l: 1, o: 0}
  {h: 0, e: 0, l: 1, o: 0}
  {h: 0, e: 0, l: 0, o: 1}
  
  ```
  
  I was wondering if one could just as well use the ASCII encoding of the text and feed the bits in one by one. So the input "hello" would be input like
  
  ```
  0110100001100101011011000110110001101111 
  
  ```
  
  Would the RNN have a disproportionately harder time having to figure out how the arbitrary and complex 8-bit ASCII encoding should be used? Or would the ASCII encoding lead to about the same performance as the nicer 1-hot encoding?
  
  **Tags:** neural-networks, machine-learning, natural-language-processing, recurrent-neural-networks, long-short-term-memory
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-09-16. Score: 2
  
  My understanding is that the ASCII encoding would not get the best performance or results from the RNN because the ASCII codes for each character are not meaningful; they are arbitrary. If the number of each ASCII code represented something meaningful about the letter, it would work better. But they don't.
  
  The same principles apply as when deciding how to encode any categorical data. If your categories are ordinal (eg. 'First', 'Second' .. or 'Age group 18-24', Age group 25-35' .. or even 'Social Class E', 'Social Class D' ..), then assigning a single numerical value to each class might work well. But in categorical data where there is no meaningful order, one hot encoding will work better.
  
  This is an example of the principle of giving neural networks the most **expressive** data that we can. In the case of non-ordinal, arbitrary categories, one-hot is more expressive to the next layer of neurons (will stimulate them more distinctly) than using a numerical encoding.
  
  
  ---
  
  ### Answer by username_2 on 2022-02-13. Score: 0
  
  The request for input encoding is that mathematical difference between encoded inputs (numerical subtraction in absolute value of vector norm, etc) must be proportional to logical (real world) dis-similitude between the objects they represent.
  
  Lets say by example the 3 first uppercase letters. In one-hot encoding they will be:
  
  ```
  'A' : (1,0,0)
  'B' : (0,1,0)
  'C' : (0,0,1)
  
  ```
  
  Note that the distance between any two pairs is 0 if the letters are the same and 2 if they differ.
  
  Compare with ASCII in binary as encoding:
  
  ```
  'A' : 0100 0001  
  'B' : 0100 0010  
  'C' : 0100 0011
  
  ```
  
  Note distance between 'A' and 'B' is 2, but distance between 'A' and 'C' is 1. Thus, we are saying that an 'A' is more similar to a 'C' than a 'B'. Something artificial that will cause network errors.
  
  Equivalent problem appears when using ASCII as integer:
  
  ```
  'A' : 65
  'B' : 66  
  'C' : 67
  
  ```
  
  we are saying that distance between 'A' and 'C' is 2, twice the distance between 'A' and 'B'. Again, something that will disturb network performance.
  
  
  ---
  
  