Answers.Body: <p>Some research areas that come to mind which can be useful when faced with a limited amount of data:</p>
  <ul>
  <li><p>Regularization: Comprises different methods to prevent the network from overfitting, to make it perform better on the validation data but not necessarily on the training data. In general, the less training data you have, the stronger you want to regularize. Common types include:</p>
  <ul>
  <li><p>Injecting noise in the network, e.g., <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf" rel="nofollow noreferrer">dropout</a>.</p>
  </li>
  <li><p>Adding regularization terms to the training loss, e.g., L1 and L2 regularization of the weights, but also <a href="https://arxiv.org/abs/1701.06548" rel="nofollow noreferrer">confident output distributions</a> can be penalized.</p>
  </li>
  <li><p>Reducing the number of parameters in the network to make it unable to fit the training data completely and thus unable to overfit badly. Interestingly, <a href="https://openai.com/blog/deep-double-descent/" rel="nofollow noreferrer"><em>increasing</em></a> the number of parameters for large models can also improve the validation performance.</p>
  </li>
  <li><p>Early stopping of training. For example, if one part of the training set is set aside and not used to update the weights, training can be stopped when the observed loss on this part of the training set is observed to start to increase.</p>
  </li>
  </ul>
  </li>
  <li><p>Generating new training data:</p>
  <ul>
  <li><p>Data augmentation: Ways to augment existing training examples without removing the semantics, e.g., slight rotations, crops, translations (shifts) of images.</p>
  </li>
  <li><p>Data interpolation, e.g., <a href="https://arxiv.org/abs/1806.05236" rel="nofollow noreferrer">manifold mixup</a>.</p>
  </li>
  <li><p>Using synthetic data, e.g., frames from video games or other CGI.</p>
  </li>
  </ul>
  </li>
  <li><p>Transfer learning: When you take a neural network that has already been trained on another, much larger dataset of the same modality (images, sounds, etc.) as your dataset and fine-tune it on your data.</p>
  </li>
  <li><p>Multitask learning: Instead of training your network to perform one task, you give it multiple output heads and train it to perform many tasks at once, given that you have that the labels for the additional tasks. While it may seem that this is a more difficult for the network, the extra tasks have a regularizing effect.</p>
  </li>
  <li><p>Semi-supervised learning: If you have much unlabeled data that labeled data, you can combine supervised learning with unsupervised learning. Much like with multitask learning, the extra task introduced by the unsupervised learning also has a regularizing effect.</p>
  </li>
  </ul>
  <p>Other interesting methods can be found in systems that perform <a href="https://arxiv.org/abs/1606.04080" rel="nofollow noreferrer">one-shot learning</a>, which inherently implies very little training data. These systems often uses slightly modified network architectures. For example, facial recognition systems can learn to recognize a face from only a single photo, and usually use a <a href="https://arxiv.org/abs/1503.03832" rel="nofollow noreferrer">triplet loss</a> (or similar) of a vector encoding of the face, instead of cross-entropy loss of the output of a softmax layer normally used for image classification.</p>
  <p>Zero-shot learning also exists (e.g., <a href="https://ai.googleblog.com/2016/11/zero-shot-translation-with-googles.html" rel="nofollow noreferrer">zero-shot machine translation</a>), but this is a completely different type of problem setup and requires multiple data modalities.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-16T23:05:30.247Z
Answers.Text: Can transfer learning be done with any type/variation of CNN architecture?
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-17T21:17:53.230Z
Answers.Text: Transfer learning can be done with any type of neural network as long as there are large enough commonalities between the dataset the network was originally trained on and your dataset, if your data is structured in the same way, as far as I know. So yes, it works for any type of CNN.
Answers.UserDisplayName: HelloGoodbye
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-16T14:41:40.057Z
Answers.LastActivityDate: 2020-09-16T14:41:40.057Z
Answers.UserDisplayName: HelloGoodbye
Body: <p>I've been reading through the research literature for image processing, computer vision, and convolutional neural networks. For image classification and object recognition, I know that convolutional neural networks deliver state-of-the-art performance when large amounts of data are available. Furthermore, I know that Hinton et al. created &quot;capsule networks&quot; to try and overcome some of the fundamental limitations of CNN architecture (such as them not being rotationally invariant). However, my understanding is that capsule networks have been a failure (so far), and most people expect them to go nowhere. And CNNs have progressively been improved in various ways (Bayesian optimisation for hyper parameter tuning, new convolution kernels, etc.). It seems to me that, at the moment, and for the foreseeable future, CNNs are the best architecture available for image-related stuff.</p>
  <p>But, as I said, CNNs, like other Deep Learning architectures, require large amounts of data. So my question is as follows:</p>
  <p>What are the research areas/topics for improving CNNs in the sense of making them work more effectively (that is, have greater performance) with less data (working with small datasets)?</p>
  <p>I know that there is various research looking at approaches to <em>increasing</em> data (such as data augmentation, generative networks, etc.), but I am primarily interested in fundamental modifications to CNNs themselves, rather than purely focusing on changes to the data itself.</p>
  <p>And to expand upon my question, using my above definition of &quot;performance&quot;, I am interested in these two categories:</p>
  <ol>
  <li><p>&quot;Computational methods&quot; for increasing CNN performance. This would be the non-mathematical stuff that I've read about, such as just increasing the number of layers and making the CNN deeper/wider (and I think another one had to do with just making the size of the convolution kernel smaller, so that it looks at smaller pieces of the image at any one time, or something like that?).</p>
  </li>
  <li><p>&quot;Mathematical methods&quot; for increasing CNN performance. This would be the cutting-edge mathematical/statistical stuff that I've read about: things like algorithms (such as Bayesian optimization); I've come across a lot of geometric stuff; and I guess the cutting-edge convolution kernels created by the image processing people would also fall under this category.</p>
  </li>
  </ol>
  <p>Obviously, this &quot;list&quot; is not exhaustive, and it's probably incorrect; I'm a novice to this research, so I'm trying to find my way around.</p>
  <p>I am interested in studying <em>both</em> of the above categories, but I will primarily be working from the mathematical/statistical side. And I want to work on research that is still <em>practical</em> and can be put to use in industry for improved performance (even if it might still be &quot;advanced&quot;/complex for most people in industry) – not the the highly theoretical stuff related.</p>
  <p>Related (but unanswered): <a href="https://ai.stackexchange.com/q/20521/16521">Are there any good research papers on image identification with limited data?</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-15T16:23:47.413Z
Comments.Text: So basically you're wondering about ways to make neural network training more data-efficient?
Comments.UserDisplayName: HelloGoodbye
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-15T16:28:37.790Z
Comments.Text: This question has extremely many answers. Basically, any method that improves the performance on neural networks without adding more data would qualify, which I would say is what the majority of papers in the field are about. Or in other words, if a method improves the performance of a neural network when faced with a lot of data, it will likely also improve the performance of a neural network when faced with little data.
Comments.UserDisplayName: HelloGoodbye
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-15T22:20:43.333Z
Comments.Text: @HelloGoodbye "data-efficient" – I think that's a good term. Yes, I suspected that that would be the case. Are there any areas that are particularly focused on, or effective at, improving "data-efficiency" in CNNs? I'm not looking for an exhaustive list – just broad categories with some examples (in particular, on the mathematics/statistics side). For someone starting in the field, it's difficult to know where to start, so I just need some information to help me along.
Comments.UserDisplayName: The Pointer
Tags: <convolutional-neural-networks><math><research><image-processing><statistics>
Title: Research paths/areas for improving the performance of CNNs when faced with limited data
UserDisplayName: The Pointer
ThreadText: # Research paths/areas for improving the performance of CNNs when faced with limited data
  
  (Asked by: username_0 on 2020-09-15)
  
  I've been reading through the research literature for image processing, computer vision, and convolutional neural networks. For image classification and object recognition, I know that convolutional neural networks deliver state-of-the-art performance when large amounts of data are available. Furthermore, I know that Hinton et al. created "capsule networks" to try and overcome some of the fundamental limitations of CNN architecture (such as them not being rotationally invariant). However, my understanding is that capsule networks have been a failure (so far), and most people expect them to go nowhere. And CNNs have progressively been improved in various ways (Bayesian optimisation for hyper parameter tuning, new convolution kernels, etc.). It seems to me that, at the moment, and for the foreseeable future, CNNs are the best architecture available for image-related stuff.
  
  But, as I said, CNNs, like other Deep Learning architectures, require large amounts of data. So my question is as follows:
  
  What are the research areas/topics for improving CNNs in the sense of making them work more effectively (that is, have greater performance) with less data (working with small datasets)?
  
  I know that there is various research looking at approaches to *increasing* data (such as data augmentation, generative networks, etc.), but I am primarily interested in fundamental modifications to CNNs themselves, rather than purely focusing on changes to the data itself.
  
  And to expand upon my question, using my above definition of "performance", I am interested in these two categories:
  
  1. "Computational methods" for increasing CNN performance. This would be the non-mathematical stuff that I've read about, such as just increasing the number of layers and making the CNN deeper/wider (and I think another one had to do with just making the size of the convolution kernel smaller, so that it looks at smaller pieces of the image at any one time, or something like that?).
  2. "Mathematical methods" for increasing CNN performance. This would be the cutting-edge mathematical/statistical stuff that I've read about: things like algorithms (such as Bayesian optimization); I've come across a lot of geometric stuff; and I guess the cutting-edge convolution kernels created by the image processing people would also fall under this category.
  
  Obviously, this "list" is not exhaustive, and it's probably incorrect; I'm a novice to this research, so I'm trying to find my way around.
  
  I am interested in studying *both* of the above categories, but I will primarily be working from the mathematical/statistical side. And I want to work on research that is still *practical* and can be put to use in industry for improved performance (even if it might still be "advanced"/complex for most people in industry) – not the the highly theoretical stuff related.
  
  Related (but unanswered): [Are there any good research papers on image identification with limited data?](https://ai.stackexchange.com/q/20521/16521)
  
  **Tags:** convolutional-neural-networks, math, research, image-processing, statistics
  
  > **Comments:**
  > **username_1**: So basically you're wondering about ways to make neural network training more data-efficient?  
  > **username_1**: This question has extremely many answers. Basically, any method that improves the performance on neural networks without adding more data would qualify, which I would say is what the majority of papers in the field are about. Or in other words, if a method improves the performance of a neural network when faced with a lot of data, it will likely also improve the performance of a neural network when faced with little data.  
  > **username_0**: @username_1 "data-efficient" – I think that's a good term. Yes, I suspected that that would be the case. Are there any areas that are particularly focused on, or effective at, improving "data-efficiency" in CNNs? I'm not looking for an exhaustive list – just broad categories with some examples (in particular, on the mathematics/statistics side). For someone starting in the field, it's difficult to know where to start, so I just need some information to help me along.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-16. Score: 2 | Accepted answer
  
  Some research areas that come to mind which can be useful when faced with a limited amount of data:
  
  * Regularization: Comprises different methods to prevent the network from overfitting, to make it perform better on the validation data but not necessarily on the training data. In general, the less training data you have, the stronger you want to regularize. Common types include:
  
    + Injecting noise in the network, e.g., [dropout](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf).
    + Adding regularization terms to the training loss, e.g., L1 and L2 regularization of the weights, but also [confident output distributions](https://arxiv.org/abs/1701.06548) can be penalized.
    + Reducing the number of parameters in the network to make it unable to fit the training data completely and thus unable to overfit badly. Interestingly, [*increasing*](https://openai.com/blog/deep-double-descent/) the number of parameters for large models can also improve the validation performance.
    + Early stopping of training. For example, if one part of the training set is set aside and not used to update the weights, training can be stopped when the observed loss on this part of the training set is observed to start to increase.
  * Generating new training data:
  
    + Data augmentation: Ways to augment existing training examples without removing the semantics, e.g., slight rotations, crops, translations (shifts) of images.
    + Data interpolation, e.g., [manifold mixup](https://arxiv.org/abs/1806.05236).
    + Using synthetic data, e.g., frames from video games or other CGI.
  * Transfer learning: When you take a neural network that has already been trained on another, much larger dataset of the same modality (images, sounds, etc.) as your dataset and fine-tune it on your data.
  * Multitask learning: Instead of training your network to perform one task, you give it multiple output heads and train it to perform many tasks at once, given that you have that the labels for the additional tasks. While it may seem that this is a more difficult for the network, the extra tasks have a regularizing effect.
  * Semi-supervised learning: If you have much unlabeled data that labeled data, you can combine supervised learning with unsupervised learning. Much like with multitask learning, the extra task introduced by the unsupervised learning also has a regularizing effect.
  
  Other interesting methods can be found in systems that perform [one-shot learning](https://arxiv.org/abs/1606.04080), which inherently implies very little training data. These systems often uses slightly modified network architectures. For example, facial recognition systems can learn to recognize a face from only a single photo, and usually use a [triplet loss](https://arxiv.org/abs/1503.03832) (or similar) of a vector encoding of the face, instead of cross-entropy loss of the output of a softmax layer normally used for image classification.
  
  Zero-shot learning also exists (e.g., [zero-shot machine translation](https://ai.googleblog.com/2016/11/zero-shot-translation-with-googles.html)), but this is a completely different type of problem setup and requires multiple data modalities.
  
  > **Comments:**
  > **username_0**: Can transfer learning be done with any type/variation of CNN architecture?  
  > **username_1**: Transfer learning can be done with any type of neural network as long as there are large enough commonalities between the dataset the network was originally trained on and your dataset, if your data is structured in the same way, as far as I know. So yes, it works for any type of CNN.  
  
  ---
  
  