Answers.Body: <p>So the training data is a small number of examples (400) drawn from a small set of fonts, and the test data is a much larger dataset drawn from a much larger set of fonts and is therefore much more variable than the training data. Two issues here are the small training data size and the difference in distributions between the training and test data. I would try the following:</p>
  <ul>
  <li>Instead of defining your own architecture, try some of the pretrained architectures available in Keras such as ResNet50 or VGG16. You can start them either with random weights or imagenet weights. Remove the top layer and put your own layers on. You can also selectively unfreeze layers and see if that makes any difference.</li>
  <li>To deal with the issues I mention above, use data augmentation to introduce variability into the training set.</li>
  </ul>
  <p>You also have not mentioned in the post how you decide to stop training the network. In case the network is overfitting, you could try finishing training earlier.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-23T19:15:04.710Z
Answers.LastActivityDate: 2021-04-23T19:15:04.710Z
Answers.UserDisplayName: Mike NZ
Body: <p>I am doing some experimentation on neural networks, and for that I am trying to program a plain <a href="https://en.wikipedia.org/wiki/Optical_character_recognition" rel="nofollow noreferrer">OCR</a> task. I have learned CNNs are the best choice ,but for the time being and due to my inexperience, I wanna go step by step and start with feedforward nets.</p>
  
  <p>So my training data is a set of roughly 400 16*16 images extracted from a script that draws every alphabet char in a tiny image for a small set of fonts registered in my computer.</p>
  
  <p>Then the test data set is extracted from the same procedure, but for all fonts in my computer.</p>
  
  <p>Well, results are quite bad. Get an accuracy of aprox. 45-50%, which is very poor... but that's not my question.</p>
  
  <p>The point is that I can't get the MSE below 0.0049, no matter what hidden layer distribution I apply to the net. I have tried with several architectures and all comes down to this figure. Does that mean the net cannot learn any further given the data?</p>
  
  <p>This MSE value however throws this poor results too.</p>
  
  <p>I am using Tensorflow API directly, no keras or estimators and for a list of 62 recognizable characters these are examples of the architectures I have used: [256,1860,62] [256,130,62] [256,256, 128 ,62] [256,3600,62] ....</p>
  
  <p>But never get the MSE below 0.0049, and still results are not over 50%.</p>
  
  <p>Any hints are greatly appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-04T22:23:52.953Z
Comments.Text: actually it's not only about architecture, how about your learning rate value? have you try to tune it? how about the number of your training epoch?
Comments.UserDisplayName: malioboro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-05T10:04:51.907Z
Comments.Text: 400 training examples is quite a small number, I think, in order to solve this task. You will need thousands of pictures in order to achieve good results, I guess.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T07:44:19.650Z
Comments.Text: No 62. The output of the net is a one-hot array, each position corresponding to a different character to recognize
Comments.UserDisplayName: Chal.lo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-24T17:18:35.960Z
Comments.Text: I would recommend using a larger dataset. The MNIST or EMNIST datasets are great for training OCRs. Also, how many generations are you training your network for.
Comments.UserDisplayName: Sean Mabli
Tags: <feedforward-neural-networks><optical-character-recognition>
Title: Attempting to solve a optical character recognition task using a feed-forward network
UserDisplayName: Chal.lo
ThreadText: # Attempting to solve a optical character recognition task using a feed-forward network
  
  (Asked by: username_0 on 2019-03-04)
  
  I am doing some experimentation on neural networks, and for that I am trying to program a plain [OCR](https://en.wikipedia.org/wiki/Optical_character_recognition) task. I have learned CNNs are the best choice ,but for the time being and due to my inexperience, I wanna go step by step and start with feedforward nets.
  
  So my training data is a set of roughly 400 16\*16 images extracted from a script that draws every alphabet char in a tiny image for a small set of fonts registered in my computer.
  
  Then the test data set is extracted from the same procedure, but for all fonts in my computer.
  
  Well, results are quite bad. Get an accuracy of aprox. 45-50%, which is very poor... but that's not my question.
  
  The point is that I can't get the MSE below 0.0049, no matter what hidden layer distribution I apply to the net. I have tried with several architectures and all comes down to this figure. Does that mean the net cannot learn any further given the data?
  
  This MSE value however throws this poor results too.
  
  I am using Tensorflow API directly, no keras or estimators and for a list of 62 recognizable characters these are examples of the architectures I have used: [256,1860,62] [256,130,62] [256,256, 128 ,62] [256,3600,62] ....
  
  But never get the MSE below 0.0049, and still results are not over 50%.
  
  Any hints are greatly appreciated.
  
  **Tags:** feedforward-neural-networks, optical-character-recognition
  
  > **Comments:**
  > **username_1**: actually it's not only about architecture, how about your learning rate value? have you try to tune it? how about the number of your training epoch?  
  > **username_2**: 400 training examples is quite a small number, I think, in order to solve this task. You will need thousands of pictures in order to achieve good results, I guess.  
  > **username_0**: No 62. The output of the net is a one-hot array, each position corresponding to a different character to recognize  
  > **username_3**: I would recommend using a larger dataset. The MNIST or EMNIST datasets are great for training OCRs. Also, how many generations are you training your network for.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2021-04-23. Score: 0
  
  So the training data is a small number of examples (400) drawn from a small set of fonts, and the test data is a much larger dataset drawn from a much larger set of fonts and is therefore much more variable than the training data. Two issues here are the small training data size and the difference in distributions between the training and test data. I would try the following:
  
  * Instead of defining your own architecture, try some of the pretrained architectures available in Keras such as ResNet50 or VGG16. You can start them either with random weights or imagenet weights. Remove the top layer and put your own layers on. You can also selectively unfreeze layers and see if that makes any difference.
  * To deal with the issues I mention above, use data augmentation to introduce variability into the training set.
  
  You also have not mentioned in the post how you decide to stop training the network. In case the network is overfitting, you could try finishing training earlier.
  
  
  ---
  
  