Answers.Body: <p>while using a neural network for this type of problem is not the ideal use-case, it is a good exercise.</p>
  <p>In terms of conceptual issues, the most concerning that I see is the loss: <span class="math-container">$\sum_{i=1}^N L_i$</span>.</p>
  <p>First issue, is that it validates loss at each time step equivalently. This is probably not ideal because in the example (cat), we dont expect it to know its English from just c or ca, but at cat. The quickest fix and probably the best, is to just use <span class="math-container">$L = L_N$</span>. Though an argument would be that the model should become more and more aware as it gets more letters, and this loss function doesnt achieve that, so another solution would be to add another fixed parameter that you can play with: <span class="math-container">$L = \sum_i r^{-i}L_i$</span> where <span class="math-container">$0 \lt r \lt 1$</span>. Note that <span class="math-container">$r^{-i}$</span> can be replaced with any function that increases in size.</p>
  <p>Another issue with the loss is it doesnt normalize, meaning on average, larger words will hold more weight to the model than smaller ones, while this may be intended it should be noted and considered (also note if you do end up going with <span class="math-container">$L=L_N$</span> this will no longer be a concern.</p>
  <p>Hope this helps</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-21T18:35:06.847Z
Answers.Text: Thank you for your answer. I just tried implementing your modification and the performance of the network seemed to improve a bit. Still not perfect, but modifying the loss function was definitely  a step in the right direction. Btw, splitting the word into characters, and feeding them one by one to the network was not meant for testing if each substring is a valid word, but if a substring is a likely sequence of characters, for example: is it probable for the letter "t" to come after the sequence "ca".
Answers.UserDisplayName: Roy Varon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-21T19:37:57.137Z
Answers.Text: are labels done that way? ex: flys -> 0011
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-21T22:30:50.530Z
Answers.Text: Almost, they are encoded using one-hot vectors. ex: flys -> [onehot(START), onehot('f'), onehot('l'), onehot('y'), onehot('s'), onehot(END)]
Answers.UserDisplayName: Roy Varon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-21T22:31:11.643Z
Answers.Text: i was asking about the labels
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-21T22:32:38.950Z
Answers.Text: I don't think I understand. There are no labels. There is just a database of English words.
Answers.UserDisplayName: Roy Varon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-20T21:17:22.123Z
Answers.LastActivityDate: 2021-03-20T21:17:22.123Z
Answers.UserDisplayName: mshlis
Answers.Body: <p>I think the problem is that you're only training the network on words. Every example in your training data has a desired label of &quot;is a word,&quot; and so your network could achieve the lowest possible loss by simply giving a probability of 100% to &quot;is a word&quot; all of the time.</p>
  <p>The most straightforward way to fix this would be to also include non-words in your training data. Of course, the words should have a target label of &quot;is a word&quot; and the non-words should have a target label of &quot;is not a word.&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T20:25:18.437Z
Answers.LastActivityDate: 2021-03-22T20:25:18.437Z
Answers.UserDisplayName: Sophie Swett
Body: <h2>Goal</h2>
  <p>To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding).</p>
  <p><strong>Example</strong></p>
  <pre><code>input:  hello 
  output: 100%
  
  input:  nmnmn 
  output: 0%
  </code></pre>
  <h2>Approach</h2>
  <p>Here is my approach.</p>
  <h3>RNN</h3>
  <p>I have built an RNN with the following specifications: (the subscript <span class="math-container">$i$</span> means a specific time step)</p>
  <p>The vectors (neurons):</p>
  <p><span class="math-container">$$
  x_i \in \mathbb{R}^n \\
  s_i \in \mathbb{R}^m \\
  h_i \in \mathbb{R}^m \\
  b_i \in \mathbb{R}^n \\
  y_i \in \mathbb{R}^n \\
  $$</span></p>
  <p>The matrices (weights):
  <span class="math-container">$$
  U \in \mathbb{R}^{m \times n} \\
  W \in \mathbb{R}^{m \times m} \\
  V \in \mathbb{R}^{n \times m} \\
  $$</span></p>
  <p>This is how each time step is being fed forward:</p>
  <p><span class="math-container">$$
  y_i = softmax(b_i) \\
  b_i = V h_i \\
  h_i = f(s_i) \\
  s_i = U x_i + W h_{i-1} \\
  $$</span>
  Note that the <span class="math-container">$ + W h_{i-1}$</span> will not be used on the first layer.</p>
  <h3>Losses</h3>
  <p>Then, for the loss of each layer, I used cross entropy (<span class="math-container">$t_i$</span> is the target, or expected output at time <span class="math-container">$i$</span>):
  <span class="math-container">$$
  L_i = -\sum_{j=1}^{n} t_{i,j} \ln(y_{i,j})
  $$</span></p>
  <p>Then, the total loss of the network:
  <span class="math-container">$$
  L = \sum L_i
  $$</span></p>
  <h3>RNN diagram</h3>
  <p>Here is a picture of the network that I drew:</p>
  <p><a href="https://i.sstatic.net/MeIA8m.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MeIA8m.png" alt="enter image description here" /></a></p>
  <h3>Data pre-processing</h3>
  <p>Here is how data is fed into the network:</p>
  <p>Each word is split into characters, and every character is split into a one-hot vector. Two special tokens START and END are being appended to the word from the beginning and the end. Then the input at each time step will be every sequential character without END, and the output at each time step will be the following character to the input.</p>
  <p><strong>Example</strong></p>
  <p>Here is an example:</p>
  <ol>
  <li>Start with a word: &quot;cat&quot;</li>
  <li>Split it into characters and append the special tags: <code>START  c  a  t  END</code></li>
  <li>Transform into one-hot vectors: <span class="math-container">$v_1, v_2, v_3, v_4, v_5$</span></li>
  <li>Then the input is <span class="math-container">$v_1, v_2, v_3, v_4$</span> and the output <span class="math-container">$v_2, v_3, v_4, v_5$</span></li>
  </ol>
  <h3>Dataset</h3>
  <p>For the dataset, I used a list of English words.</p>
  <p>Since I am working with English characters, the size of the input and output is <span class="math-container">$n=26+2=28$</span> (the <span class="math-container">$+2$</span> is for the extra START and END tags).</p>
  <h3>Hyper-parameters</h3>
  <p>Here are some more specifications:</p>
  <ul>
  <li>Hidden size: <span class="math-container">$m=100$</span></li>
  <li>Learning rate:  <span class="math-container">$0.001$</span></li>
  <li>Number of training cycles:  <span class="math-container">$15000$</span> (each cycle is a loss calculation and backpropagation of a random word)</li>
  <li>Activation function: <span class="math-container">$f(x) = \tanh(x)$</span></li>
  </ul>
  <h2>Problem/question</h2>
  <p>However, when I run my model, I get that the probability of some word being valid is about 0.9 regardless of the input.</p>
  <p>For the probability of a word begin valid, I used the value at the last layer of the RNN at the position of END tag after feeding forward the word.</p>
  <p>I wrote a gradient checking algorithm and the gradients seem to check up.</p>
  <p><em>Is there conceptually something wrong with my neural network?</em></p>
  <p>I played a bit with <span class="math-container">$m$</span>, the learning rate, and the number of cycles, but nothing really improved the performance.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-22T18:48:07.817Z
Comments.Text: Does the training data include non-words too, or only words?
Comments.UserDisplayName: Sophie Swett
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-22T19:55:53.120Z
Comments.Text: Only words. I saw a similar RNN online that used a database of sentences to generate new ones, and they didn't use "fake sentences", so I thought using only real words in mine would be enough.
Comments.UserDisplayName: Roy Varon
Tags: <neural-networks><natural-language-processing><recurrent-neural-networks><data-preprocessing>
Title: Is my approach to building an RNN to predict the probability that the word is in English appropriate?
UserDisplayName: Roy Varon
ThreadText: # Is my approach to building an RNN to predict the probability that the word is in English appropriate?
  
  (Asked by: username_0 on 2021-03-18)
  
  ## Goal
  
  To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding).
  
  **Example**
  
  ```
  input:  hello 
  output: 100%
  
  input:  nmnmn 
  output: 0%
  
  ```
  
  ## Approach
  
  Here is my approach.
  
  ### RNN
  
  I have built an RNN with the following specifications: (the subscript $i$ means a specific time step)
  
  The vectors (neurons):
  
  $$
  x\_i \in \mathbb{R}^n \\
  s\_i \in \mathbb{R}^m \\
  h\_i \in \mathbb{R}^m \\
  b\_i \in \mathbb{R}^n \\
  y\_i \in \mathbb{R}^n \\
  $$
  
  The matrices (weights):
  $$
  U \in \mathbb{R}^{m \times n} \\
  W \in \mathbb{R}^{m \times m} \\
  V \in \mathbb{R}^{n \times m} \\
  $$
  
  This is how each time step is being fed forward:
  
  $$
  y\_i = softmax(b\_i) \\
  b\_i = V h\_i \\
  h\_i = f(s\_i) \\
  s\_i = U x\_i + W h\_{i-1} \\
  $$
  Note that the $ + W h\_{i-1}$ will not be used on the first layer.
  
  ### Losses
  
  Then, for the loss of each layer, I used cross entropy ($t\_i$ is the target, or expected output at time $i$):
  $$
  L\_i = -\sum\_{j=1}^{n} t\_{i,j} \ln(y\_{i,j})
  $$
  
  Then, the total loss of the network:
  $$
  L = \sum L\_i
  $$
  
  ### RNN diagram
  
  Here is a picture of the network that I drew:
  
  [![enter image description here](https://i.sstatic.net/MeIA8m.png)](https://i.sstatic.net/MeIA8m.png)
  
  ### Data pre-processing
  
  Here is how data is fed into the network:
  
  Each word is split into characters, and every character is split into a one-hot vector. Two special tokens START and END are being appended to the word from the beginning and the end. Then the input at each time step will be every sequential character without END, and the output at each time step will be the following character to the input.
  
  **Example**
  
  Here is an example:
  
  1. Start with a word: "cat"
  2. Split it into characters and append the special tags: `START c a t END`
  3. Transform into one-hot vectors: $v\_1, v\_2, v\_3, v\_4, v\_5$
  4. Then the input is $v\_1, v\_2, v\_3, v\_4$ and the output $v\_2, v\_3, v\_4, v\_5$
  
  ### Dataset
  
  For the dataset, I used a list of English words.
  
  Since I am working with English characters, the size of the input and output is $n=26+2=28$ (the $+2$ is for the extra START and END tags).
  
  ### Hyper-parameters
  
  Here are some more specifications:
  
  * Hidden size: $m=100$
  * Learning rate: $0.001$
  * Number of training cycles: $15000$ (each cycle is a loss calculation and backpropagation of a random word)
  * Activation function: $f(x) = \tanh(x)$
  
  ## Problem/question
  
  However, when I run my model, I get that the probability of some word being valid is about 0.9 regardless of the input.
  
  For the probability of a word begin valid, I used the value at the last layer of the RNN at the position of END tag after feeding forward the word.
  
  I wrote a gradient checking algorithm and the gradients seem to check up.
  
  *Is there conceptually something wrong with my neural network?*
  
  I played a bit with $m$, the learning rate, and the number of cycles, but nothing really improved the performance.
  
  **Tags:** neural-networks, natural-language-processing, recurrent-neural-networks, data-preprocessing
  
  > **Comments:**
  > **username_1**: Does the training data include non-words too, or only words?  
  > **username_0**: Only words. I saw a similar RNN online that used a database of sentences to generate new ones, and they didn't use "fake sentences", so I thought using only real words in mine would be enough.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-03-20. Score: 0 | Accepted answer
  
  while using a neural network for this type of problem is not the ideal use-case, it is a good exercise.
  
  In terms of conceptual issues, the most concerning that I see is the loss: $\sum\_{i=1}^N L\_i$.
  
  First issue, is that it validates loss at each time step equivalently. This is probably not ideal because in the example (cat), we dont expect it to know its English from just c or ca, but at cat. The quickest fix and probably the best, is to just use $L = L\_N$. Though an argument would be that the model should become more and more aware as it gets more letters, and this loss function doesnt achieve that, so another solution would be to add another fixed parameter that you can play with: $L = \sum\_i r^{-i}L\_i$ where $0 \lt r \lt 1$. Note that $r^{-i}$ can be replaced with any function that increases in size.
  
  Another issue with the loss is it doesnt normalize, meaning on average, larger words will hold more weight to the model than smaller ones, while this may be intended it should be noted and considered (also note if you do end up going with $L=L\_N$ this will no longer be a concern.
  
  Hope this helps
  
  > **Comments:**
  > **username_0**: Thank you for your answer. I just tried implementing your modification and the performance of the network seemed to improve a bit. Still not perfect, but modifying the loss function was definitely  a step in the right direction. Btw, splitting the word into characters, and feeding them one by one to the network was not meant for testing if each substring is a valid word, but if a substring is a likely sequence of characters, for example: is it probable for the letter "t" to come after the sequence "ca".  
  > **username_2**: are labels done that way? ex: flys -> 0011  
  > **username_0**: Almost, they are encoded using one-hot vectors. ex: flys -> [onehot(START), onehot('f'), onehot('l'), onehot('y'), onehot('s'), onehot(END)]  
  > **username_2**: i was asking about the labels  
  > **username_0**: I don't think I understand. There are no labels. There is just a database of English words.  
  
  ---
  
  ### Answer by username_1 on 2021-03-22. Score: 0
  
  I think the problem is that you're only training the network on words. Every example in your training data has a desired label of "is a word," and so your network could achieve the lowest possible loss by simply giving a probability of 100% to "is a word" all of the time.
  
  The most straightforward way to fix this would be to also include non-words in your training data. Of course, the words should have a target label of "is a word" and the non-words should have a target label of "is not a word."
  
  
  ---
  
  