Answers.Body: <p>ResNets were introduced to solve the fact that a <span class="math-container">$K$</span> layer network would work worse than a <span class="math-container">$K'&lt;K$</span> layer network, and the argument was that the last <span class="math-container">$K-K'$</span> were not initialized to learn the identity function, thus the residual connection</p>
  <p>For shallow network, most likely this will not make any difference (also because probably the problem you are solving is simpler)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-11T11:21:29.727Z
Answers.Text: If I have a constraint on training time and resources, but want to train a complex problem, then which network is recommended?
Answers.UserDisplayName: ananya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-11T11:25:35.573Z
Answers.Text: @ananya that's mostly on you, the only way of making the training efficient is using inductive biases, no free lunch theorem already tells you that if you don't assume anything, than any model will perform like any other on average
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-11T11:16:24.177Z
Answers.LastActivityDate: 2024-11-11T11:16:24.177Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>If a model doesn't have many layers such as 20+ or so, it generally won't benefit from the use of a ResNet over a regular CNN, and transformer with residue connections may have hundreds of layers. In very deep networks the gradients are likely vanishing making optimization difficult. By preserving the gradient flow between layers Res-Net model makes learning the residue error easier during training.</p>
  <p>For shallow models typically with less than 10 layers, vanishing gradients are much less of an issue. In these cases, the regular CNN without residual connections is usually sufficient and can perform equally well or better, without complicated architecture and overhead like Res-Net to slow down training.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-12T06:57:55.677Z
Answers.LastActivityDate: 2024-11-12T06:57:55.677Z
Answers.UserDisplayName: cinch
Body: <p>I want to know, if the model doesn't have many layers, then will it be useful to use a ResNet or the performance will be same as using a CNN?</p>
  
Tags: <deep-learning><residual-networks>
Title: Is using a Res-Net model recommended when the model is not very deep?
UserDisplayName: ananya
ThreadText: # Is using a Res-Net model recommended when the model is not very deep?
  
  (Asked by: username_0 on 2024-11-11)
  
  I want to know, if the model doesn't have many layers, then will it be useful to use a ResNet or the performance will be same as using a CNN?
  
  **Tags:** deep-learning, residual-networks
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-11-11. Score: 0
  
  ResNets were introduced to solve the fact that a $K$ layer network would work worse than a $K' layer network, and the argument was that the last $K-K'$ were not initialized to learn the identity function, thus the residual connection
  
  For shallow network, most likely this will not make any difference (also because probably the problem you are solving is simpler)
  
  > **Comments:**
  > **username_0**: If I have a constraint on training time and resources, but want to train a complex problem, then which network is recommended?  
  > **username_1**: @ananya that's mostly on you, the only way of making the training efficient is using inductive biases, no free lunch theorem already tells you that if you don't assume anything, than any model will perform like any other on average  
  
  ---
  
  ### Answer by username_2 on 2024-11-12. Score: 0
  
  If a model doesn't have many layers such as 20+ or so, it generally won't benefit from the use of a ResNet over a regular CNN, and transformer with residue connections may have hundreds of layers. In very deep networks the gradients are likely vanishing making optimization difficult. By preserving the gradient flow between layers Res-Net model makes learning the residue error easier during training.
  
  For shallow models typically with less than 10 layers, vanishing gradients are much less of an issue. In these cases, the regular CNN without residual connections is usually sufficient and can perform equally well or better, without complicated architecture and overhead like Res-Net to slow down training.
  
  
  ---
  
  