Answers.Body: <blockquote>
  <p>Can residual connections be beneficial when we have a small training dataset?</p>
  </blockquote>
  <p>The usual rule of data science investigations applies here: Try it, measure the results, then you will know.</p>
  <p>It is very hard to tell, <em>a priori</em>, whether a specific architectural or hyperparameter choice will impact the performance of a neural network on a given problem.</p>
  <p>In this case, you are wondering whether <a href="https://theaisummer.com/skip-connections/" rel="nofollow noreferrer">residual networks using skip connections</a> might help when you have a relatively low amount of training data.</p>
  <ul>
  <li><p>On the pro side, effects of skip connections that help correct vanishing gradients, and treat each block as learning the difference from an identity function, will still work for your problem. That means you will have some freedom to explore adding layers without worrying about the negative impacts of doing so.</p>
  </li>
  <li><p>On the con side, it is unlikley that you will benefit from very deep networks as there will not be enough examples to learn truly complex functions from.</p>
  </li>
  </ul>
  <p>You may find that adding depth, but reducing &quot;width&quot;, i.e. the number of artificial neurons in each layer, will work.</p>
  <p>If you have a low amount of training data and a difficult problem to solve, then residual networks are not a magic fix. The best you could hope for is a relatively stable statistical model that works on the simpler differences between your training examples. However, it may be possible that tuning a neural network by searching through different architectures will be a worthwhile exercise.</p>
  <p>I would also suggest that for one of the good networks that you vary the number of examples used to train the network in multiple training runs, and plot a learning graph showing number of examples versus accuracy (or other metric that you may be interested in). This graph will help you decide whether collecting more training data would be worthwhile because you will have a rough estimate of the gradient for how much new training examples could improve your results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-04T10:03:00.550Z
Answers.LastActivityDate: 2021-04-05T13:32:49.703Z
Answers.LastEditDate: 2021-04-05T13:32:49.703Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have a classification problem, for which an inadequate amount of training data is available. Also, there is no known practical data augmentation approach for this problem (as no unlabelled data is available either), but I am working on it.</p>
  <p>As we know, deep neural networks require a large amount of data for training, especially when a deep architecture with many layers is used. Using these complex architectures with less data can easily lead to over-fitting. Residual connections can shortcut some blocks or layers, which can result in simpler models, while we have the benefit of complex structures.</p>
  <p>Can residual connections be beneficial when we have a small training dataset?</p>
  
Tags: <deep-learning><convolutional-neural-networks><datasets><overfitting><residual-networks>
Title: Can residual connections be beneficial when we have a small training dataset?
UserDisplayName: Reza_va
ThreadText: # Can residual connections be beneficial when we have a small training dataset?
  
  (Asked by: username_0 on 2021-04-04)
  
  I have a classification problem, for which an inadequate amount of training data is available. Also, there is no known practical data augmentation approach for this problem (as no unlabelled data is available either), but I am working on it.
  
  As we know, deep neural networks require a large amount of data for training, especially when a deep architecture with many layers is used. Using these complex architectures with less data can easily lead to over-fitting. Residual connections can shortcut some blocks or layers, which can result in simpler models, while we have the benefit of complex structures.
  
  Can residual connections be beneficial when we have a small training dataset?
  
  **Tags:** deep-learning, convolutional-neural-networks, datasets, overfitting, residual-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-04. Score: 3 | Accepted answer
  
  > Can residual connections be beneficial when we have a small training dataset?
  
  The usual rule of data science investigations applies here: Try it, measure the results, then you will know.
  
  It is very hard to tell, *a priori*, whether a specific architectural or hyperparameter choice will impact the performance of a neural network on a given problem.
  
  In this case, you are wondering whether [residual networks using skip connections](https://theaisummer.com/skip-connections/) might help when you have a relatively low amount of training data.
  
  * On the pro side, effects of skip connections that help correct vanishing gradients, and treat each block as learning the difference from an identity function, will still work for your problem. That means you will have some freedom to explore adding layers without worrying about the negative impacts of doing so.
  * On the con side, it is unlikley that you will benefit from very deep networks as there will not be enough examples to learn truly complex functions from.
  
  You may find that adding depth, but reducing "width", i.e. the number of artificial neurons in each layer, will work.
  
  If you have a low amount of training data and a difficult problem to solve, then residual networks are not a magic fix. The best you could hope for is a relatively stable statistical model that works on the simpler differences between your training examples. However, it may be possible that tuning a neural network by searching through different architectures will be a worthwhile exercise.
  
  I would also suggest that for one of the good networks that you vary the number of examples used to train the network in multiple training runs, and plot a learning graph showing number of examples versus accuracy (or other metric that you may be interested in). This graph will help you decide whether collecting more training data would be worthwhile because you will have a rough estimate of the gradient for how much new training examples could improve your results.
  
  
  ---
  
  