Answers.Body: <blockquote>
    <p>an epoch takes 3.5 days  </p>
  </blockquote>
  
  <p>First of all, use <a href="http://colab.research.google.com" rel="nofollow noreferrer">colab</a> to iterate quickly, it offers unlimited 12 hours of free GPU.  </p>
  
  <blockquote>
    <p>to retrain it on Imagenet.   </p>
  </blockquote>
  
  <p>That said, we could use complex models without being afraid of overfitting due to large training size.</p>
  
  <blockquote>
    <p>will continued training increases the accuracy of the training set at the expense of the validation set  </p>
  </blockquote>
  
  <p>In many cases that's the case.</p>
  
  <blockquote>
    <p>The idea was to mimic the massive feedback projections in the brain.  </p>
  </blockquote>
  
  <p>My suggestion is to read the third part of the <a href="http://www.deeplearningbook.org/" rel="nofollow noreferrer">deeplearning book</a> which includes Representation Learning, Structured Probabilistic Models for Deep Learning, Monte Carlo Methods, Confronting the Partition Function, Approximate Inference, Deep Generative Models.</p>
  
  <blockquote>
    <p>Part III is the most important for a researcher—someone who wants to understand
    the breadth of perspectives that have been brought to the field of deep
    learning, and push the field forward towards true artificial intelligence.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-06T23:11:04.720Z
Answers.Text: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction
Answers.UserDisplayName: x89
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-18T15:37:59.387Z
Answers.Text: thanks for the book suggestion, I will read it. I was not sure to understand your answer. Large data should not result on over fitting, but that is what I get. So could continued training improve the validation accuracy or not (it is lower after training for an epoch?). Also I have a 1080, it should be about the same speed as the k80 used in colab, right?
Answers.UserDisplayName: Wolphram jonny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-21T19:29:51.353Z
Answers.Text: The 1050ti should be the same speed as the k80, Large data should not result on overfitting BUT what if you have a very very very large model? "with a second net after the first" even with non trainable parameters. by the way, your approach is somehow ill-defined. if you insist to work on some crazy idea, consider toy problems with small (synthetic) dataset and iterate from there. All researcher try their experiments with a suitable computing/time budget.
Answers.UserDisplayName: Fadi Bakoura
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-18T15:12:15.290Z
Answers.LastActivityDate: 2018-05-18T15:12:15.290Z
Answers.UserDisplayName: Fadi Bakoura
Body: <p>I am trying a modification of Mobilenet in which I add feedback from the softmax layer into the early layers (to implement this I put a second net after the first, which receives connections from the softmax layer of the first, the pretrained weights being non trainable). The idea was to mimic the massive feedback projections in the brain, which presumably could help object recognition by enhancing specific filters and inhibiting others. </p>
  
  <p>I took the pretrained network from Keras and started to retrain it on Imagenet. I noticed that the training accuraccy increased right in the first epoch. My computer is very slow thus I cannot train for too long, an epoch takes 3.5 days. So after an epoch I tried the validation set, but instead the accuracy went down to almost half that of the pretrained values.</p>
  
  <p>My question is if this is and obvious case of overfitting. That is, will continued training increase the accuracy of the training set at the expense of the validation set, or is this a normal behavior expected at the initial stages of training, so that if I keep training for a few more epochs I could expect the validation set accuracy to go eventually up? Any ideas that could help are welcomed. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-06T23:10:51.173Z
Comments.Text: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction
Comments.UserDisplayName: x89
Tags: <convolutional-neural-networks><image-recognition><overfitting>
Title: Is this overfitting avoidable?
UserDisplayName: Wolphram jonny
ThreadText: # Is this overfitting avoidable?
  
  (Asked by: username_0 on 2018-05-15)
  
  I am trying a modification of Mobilenet in which I add feedback from the softmax layer into the early layers (to implement this I put a second net after the first, which receives connections from the softmax layer of the first, the pretrained weights being non trainable). The idea was to mimic the massive feedback projections in the brain, which presumably could help object recognition by enhancing specific filters and inhibiting others.
  
  I took the pretrained network from Keras and started to retrain it on Imagenet. I noticed that the training accuraccy increased right in the first epoch. My computer is very slow thus I cannot train for too long, an epoch takes 3.5 days. So after an epoch I tried the validation set, but instead the accuracy went down to almost half that of the pretrained values.
  
  My question is if this is and obvious case of overfitting. That is, will continued training increase the accuracy of the training set at the expense of the validation set, or is this a normal behavior expected at the initial stages of training, so that if I keep training for a few more epochs I could expect the validation set accuracy to go eventually up? Any ideas that could help are welcomed.
  
  **Tags:** convolutional-neural-networks, image-recognition, overfitting
  
  > **Comments:**
  > **username_1**: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-05-18. Score: 1 | Accepted answer
  
  > an epoch takes 3.5 days
  
  First of all, use [colab](http://colab.research.google.com) to iterate quickly, it offers unlimited 12 hours of free GPU.
  
  > to retrain it on Imagenet.
  
  That said, we could use complex models without being afraid of overfitting due to large training size.
  
  > will continued training increases the accuracy of the training set at the expense of the validation set
  
  In many cases that's the case.
  
  > The idea was to mimic the massive feedback projections in the brain.
  
  My suggestion is to read the third part of the [deeplearning book](http://www.deeplearningbook.org/) which includes Representation Learning, Structured Probabilistic Models for Deep Learning, Monte Carlo Methods, Confronting the Partition Function, Approximate Inference, Deep Generative Models.
  
  > Part III is the most important for a researcher—someone who wants to understand
  > the breadth of perspectives that have been brought to the field of deep
  > learning, and push the field forward towards true artificial intelligence.
  
  > **Comments:**
  > **username_0**: thanks for the book suggestion, I will read it. I was not sure to understand your answer. Large data should not result on over fitting, but that is what I get. So could continued training improve the validation accuracy or not (it is lower after training for an epoch?). Also I have a 1080, it should be about the same speed as the k80 used in colab, right?  
  > **username_2**: The 1050ti should be the same speed as the k80, Large data should not result on overfitting BUT what if you have a very very very large model? "with a second net after the first" even with non trainable parameters. by the way, your approach is somehow ill-defined. if you insist to work on some crazy idea, consider toy problems with small (synthetic) dataset and iterate from there. All researcher try their experiments with a suitable computing/time budget.  
  > **username_1**: Hi could you take a look here? https://ai.stackexchange.com/questions/27687/visualisation-for-mobilenet-feature-extraction  
  
  ---
  
  