Answers.Body: <p>Your reasoning isn't wrong. Deep Neural Networks (DNNs) have a much larger capacity than simpler ML algorithms (excluding NNs) and can easily memorize even a very complex dataset and overfit. </p>
  
  <p>DNNs, however, are so effective because they usually are applied on tasks that are <strong>harder</strong>, so it's not as easy to overfit. For example an image classifier might be trained on a dataset with millions of images; a task much harder to overfit on.</p>
  
  <p>In cases where this isn't possible (e.g. an image classification task with a couple thousand images), <strong>transfer learning</strong> is used. You can initialize your weights from a model pre-trained on a large dataset, use its already-trained feature extraction layers and simply fine tune the last layer.</p>
  
  <p><strong>Data augmentation</strong> also helps a lot here, which effectively increases size of the training set and discourages the DNN from memorizing the samples. It is so effective that it is used even in large datasets, where it is harder to overfit.</p>
  
  <p>Additionally, DNNs employ several methods to prevent them from overfitting. The most prominent of these is <strong>dropout</strong>, which is a very effective regularizer. <strong>Batch Normalization</strong> has also proven an effective regularizer. <strong>SGD</strong> allows you to explore more parameters than GD, which also is effective against overfitting. Finally, <strong>early stopping</strong> and <strong>parameter norm penalties</strong> aren't uncommon in DNNs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-09T06:26:11.313Z
Answers.LastActivityDate: 2019-11-09T06:26:11.313Z
Answers.UserDisplayName: Djib2011
Body: <p>In my opinion, deep learning algorithms and models (that is, multi-layer neural networks) are more sensitive to overfitting than machine learning algorithms and models (such as the SVM, random forest, perceptron, Markov models, etc.). They are capable of learning more complex patterns. At least that's how I look at it. Still, I and my colleagues disagree about this and I cannot really find any information about this. My colleagues say that deep learning algorithms are hardly vulnerable to overfitting.</p>
  
  <p>Are there statements (or opinions) about this aspect?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-08T23:56:23.053Z
Comments.Text: I've not seen significant research done on this, as it is an unusual and specific thing to compare between these models. But at a glance, I would agree that a CNN for example would be much more prone to over-fitting than an SVM, especially without the significant effort put in to ensure it doesn't over fit (dropout, regularisation and so on).
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-05T16:59:09.383Z
Comments.Text: Deep learning is a subset of machine learning.
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-14T23:06:58.343Z
Comments.Text: It might be better to view this as a problem of data; traditional methods are simpler and, while they can overfit, the techniques used do not reuse the training data repeatedly. Deep learning, however, is capable of modeling much more complex relationships and tends to require a great deal more data; with insufficient data, we are forced to pass the same data through multiple training epochs which directly increases the likelihood of overfiiting.
Comments.UserDisplayName: David Hoelzer
Tags: <machine-learning><deep-learning><comparison><overfitting>
Title: Are deep learning models more prone to overfitting than machine learning ones?
UserDisplayName: jennifer ruurs
ThreadText: # Are deep learning models more prone to overfitting than machine learning ones?
  
  (Asked by: username_0 on 2019-11-08)
  
  In my opinion, deep learning algorithms and models (that is, multi-layer neural networks) are more sensitive to overfitting than machine learning algorithms and models (such as the SVM, random forest, perceptron, Markov models, etc.). They are capable of learning more complex patterns. At least that's how I look at it. Still, I and my colleagues disagree about this and I cannot really find any information about this. My colleagues say that deep learning algorithms are hardly vulnerable to overfitting.
  
  Are there statements (or opinions) about this aspect?
  
  **Tags:** machine-learning, deep-learning, comparison, overfitting
  
  > **Comments:**
  > **username_1**: I've not seen significant research done on this, as it is an unusual and specific thing to compare between these models. But at a glance, I would agree that a CNN for example would be much more prone to over-fitting than an SVM, especially without the significant effort put in to ensure it doesn't over fit (dropout, regularisation and so on).  
  > **username_2**: Deep learning is a subset of machine learning.  
  > **username_3**: It might be better to view this as a problem of data; traditional methods are simpler and, while they can overfit, the techniques used do not reuse the training data repeatedly. Deep learning, however, is capable of modeling much more complex relationships and tends to require a great deal more data; with insufficient data, we are forced to pass the same data through multiple training epochs which directly increases the likelihood of overfiiting.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2019-11-09. Score: 5 | Accepted answer
  
  Your reasoning isn't wrong. Deep Neural Networks (DNNs) have a much larger capacity than simpler ML algorithms (excluding NNs) and can easily memorize even a very complex dataset and overfit.
  
  DNNs, however, are so effective because they usually are applied on tasks that are **harder**, so it's not as easy to overfit. For example an image classifier might be trained on a dataset with millions of images; a task much harder to overfit on.
  
  In cases where this isn't possible (e.g. an image classification task with a couple thousand images), **transfer learning** is used. You can initialize your weights from a model pre-trained on a large dataset, use its already-trained feature extraction layers and simply fine tune the last layer.
  
  **Data augmentation** also helps a lot here, which effectively increases size of the training set and discourages the DNN from memorizing the samples. It is so effective that it is used even in large datasets, where it is harder to overfit.
  
  Additionally, DNNs employ several methods to prevent them from overfitting. The most prominent of these is **dropout**, which is a very effective regularizer. **Batch Normalization** has also proven an effective regularizer. **SGD** allows you to explore more parameters than GD, which also is effective against overfitting. Finally, **early stopping** and **parameter norm penalties** aren't uncommon in DNNs.
  
  
  ---
  
  