Answers.Body: <p>The main reason of overfitting in any neural network is having too many unrestricted trainable degrees of freedom in the model. Methods similar to dropout reduce the number of neurons at each training run which effectively means having a smaller network. On the other hand in <span class="math-container">$l_1$</span>  and <span class="math-container">$l_2$</span> regularization, a term added to the loss function which put a constraint on the total loss calculated at each run. So what we are trying to minimize with such regularizations is not just <span class="math-container">$L$</span>, but <span class="math-container">$L +l_1*f(w)$</span> (for example). </p>
  
  <p>What I understand from that paper is that the auxiliary outputs do both at the same time: the point is to build smaller version of the same inception network within itself, and use the losses obtained from those as a constraint on the final loss function. The full model described in the paper is essentially 3 separate models: the first one is a network with 3 inception modules, the second one with 6 and the final one has 9. When loss is calculated, results from the auxiliary outputs added to the total loss with a 0.3 weight. Let us write this as follows:</p>
  
  <p><span class="math-container">$L = 0.7*L_9+ 0.3*(L_6+L_3).  $</span></p>
  
  <p>Here <span class="math-container">$L_3$</span> and <span class="math-container">$L_6$</span> represent the losses calculated at first and second outputs, and <span class="math-container">$L_9$</span> is the loss calculated at the final output of the network.</p>
  
  <p>This is the function we wish to minimize during training. But when the evaluation is made, the auxiliary outputs are discarded, and just the final softmax layer is used. Not very dissimilar from the idea of using dropout during training but using the full model for predictions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-01T02:27:53.480Z
Answers.Text: It might be a good idea to use latex to format your answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-30T09:05:01.503Z
Answers.LastActivityDate: 2019-12-01T06:03:32.247Z
Answers.LastEditDate: 2019-12-01T06:03:32.247Z
Answers.UserDisplayName: serali
Body: <p>Today I was going through a tutorial of Andrew Ng about <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43022.pdf" rel="nofollow noreferrer">Inception network</a>. He said that <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43022.pdf" rel="nofollow noreferrer">GoogLeNet</a>'s hidden layers are also good in prediction and it had somehow a regularization effect, so it reduces overfitting. I also search on this topic and tried to figure out by reading the <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43022.pdf" rel="nofollow noreferrer">GoogLeNet paper</a>. But I am not satisfied. </p>
  
  <p>Can anyone give me any mathematical intuition or reasoning about this in detail?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-29T21:33:45.017Z
Comments.Text: Hi. Please, add a link to the tutorial you were reading or watching.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-29T21:59:12.077Z
Comments.Text: Course: Convolutional Neural Network, Vendor: deeplearning.ai, Website: coursera.org, Course Timestamp: 2nd week (InceptionNet)
Comments.UserDisplayName: Mahir Mahbub
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-29T22:37:48.037Z
Comments.Text: Is [this](https://www.youtube.com/watch?v=C86ZXvgpejM) the video?
Comments.UserDisplayName: nbro
Tags: <neural-networks><convolutional-neural-networks><papers><overfitting>
Title: How GoogleNet actually deal with reducing overfitting?
UserDisplayName: Mahir Mahbub
ThreadText: # How GoogleNet actually deal with reducing overfitting?
  
  (Asked by: username_0 on 2019-11-29)
  
  Today I was going through a tutorial of Andrew Ng about [Inception network](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43022.pdf). He said that [GoogLeNet](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43022.pdf)'s hidden layers are also good in prediction and it had somehow a regularization effect, so it reduces overfitting. I also search on this topic and tried to figure out by reading the [GoogLeNet paper](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43022.pdf). But I am not satisfied.
  
  Can anyone give me any mathematical intuition or reasoning about this in detail?
  
  **Tags:** neural-networks, convolutional-neural-networks, papers, overfitting
  
  > **Comments:**
  > **username_1**: Hi. Please, add a link to the tutorial you were reading or watching.  
  > **username_0**: Course: Convolutional Neural Network, Vendor: deeplearning.ai, Website: coursera.org, Course Timestamp: 2nd week (InceptionNet)  
  > **username_1**: Is [this](https://www.youtube.com/watch?v=C86ZXvgpejM) the video?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-11-30. Score: 1
  
  The main reason of overfitting in any neural network is having too many unrestricted trainable degrees of freedom in the model. Methods similar to dropout reduce the number of neurons at each training run which effectively means having a smaller network. On the other hand in $l\_1$ and $l\_2$ regularization, a term added to the loss function which put a constraint on the total loss calculated at each run. So what we are trying to minimize with such regularizations is not just $L$, but $L +l\_1\*f(w)$ (for example).
  
  What I understand from that paper is that the auxiliary outputs do both at the same time: the point is to build smaller version of the same inception network within itself, and use the losses obtained from those as a constraint on the final loss function. The full model described in the paper is essentially 3 separate models: the first one is a network with 3 inception modules, the second one with 6 and the final one has 9. When loss is calculated, results from the auxiliary outputs added to the total loss with a 0.3 weight. Let us write this as follows:
  
  $L = 0.7\*L\_9+ 0.3\*(L\_6+L\_3). $
  
  Here $L\_3$ and $L\_6$ represent the losses calculated at first and second outputs, and $L\_9$ is the loss calculated at the final output of the network.
  
  This is the function we wish to minimize during training. But when the evaluation is made, the auxiliary outputs are discarded, and just the final softmax layer is used. Not very dissimilar from the idea of using dropout during training but using the full model for predictions.
  
  > **Comments:**
  > **username_1**: It might be a good idea to use latex to format your answer.  
  
  ---
  
  