Answers.Body: <p>For the multiclass SVM, there will be an ensembling effect since you are learning 5*4=20 1vs1 classifiers. It could be an interesting experiment to try the same thing with simple neural networks. Also, since you are standardizing the inputs you could try tanh activations on the first layer after the input. I presume you are using softmax on the output layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-21T04:54:42.600Z
Answers.LastActivityDate: 2021-04-21T04:54:42.600Z
Answers.UserDisplayName: Mike NZ
Answers.Body: <p>I guess you are using linear activation functions,  maybe you are not initializing your weights, or you are regularizing your model enough.  </p>
  
  <p>Initialize weights with <a href="http://www.jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf?hc_location=ufi" rel="nofollow noreferrer">glorot</a>, insert dropout layers in between, use Relu as your activation function, stop the training process based on Early Stopping. and just experiment with one hidden layer.</p>
  
  <p>side note: if you use Adam, don't mess with the learning rate.<br>
  in SGD optimizer you could use decay because there is a single learning rate for all weight updates and the learning rate does not change during training.<br>
  In Adam, the learning rate is maintained for each network weight (parameter) and separately adopted as learning unfolds. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-06T16:34:13.743Z
Answers.Text: I'm using pytorch to model MLP. I'm assuming weight initialization is handled by pytorch. Also, I'm using ReLU and regularization is 0 by default in pytorch.
  
  Also, what sense does it make to use dropout layer when I'm not able to overfit on train data?
  
  I'll look into the learning rate aspect you mentioned.
Answers.UserDisplayName: Ashwin Kannan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-06T16:00:36.640Z
Answers.LastActivityDate: 2018-05-06T16:00:36.640Z
Answers.UserDisplayName: Fadi Bakoura
Body: <p>I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples). </p>
  
  <p>With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.</p>
  
  <p>Now moving on to MLP results,</p>
  
  <p><strong>Exp 1:</strong></p>
  
  <ul>
  <li><em>Network Architecture:</em> [67 40 5]</li>
  <li><em>Optimizer:</em> Adam</li>
  <li><em>Learning Rate:</em> exponential decay of base learning rate</li>
  <li><em>Validation Accuracy:</em> ~45%</li>
  <li><em>Observation:</em> Both training accuracy and validation accuracy stops improving.</li>
  </ul>
  
  <p><strong>Exp 2:</strong>
  Repeated <strong>Exp 1</strong> with batchnorm layer</p>
  
  <ul>
  <li><em>Validation Accuracy:</em> ~50%</li>
  <li><em>Observation:</em> Got 5% increase in accuracy.</li>
  </ul>
  
  <p><strong>Exp 3:</strong></p>
  
  <p>To overfit, increased the depth of MLP. A deeper version of <strong>Exp 1</strong> network</p>
  
  <ul>
  <li><em>Network Architecture:</em> [67 40 40 40 40 40 40 5]</li>
  <li><em>Optimizer:</em> Adam</li>
  <li><em>Learning Rate:</em> exponential decay of base learning rate</li>
  <li><em>Validation Accuracy:</em> ~55%</li>
  </ul>
  
  <p>Thoughts on what might be happening? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-28T08:13:04.863Z
Comments.Text: Such results are not unexpected. 67 dim vector is small, multiclass SVM would likely produce better results then NN. NN start to shine with dimensionality >=1000. If you actually want overfit, just for fun, you probably need bigger network with 128 to 512 layer size, at least 3 layers (not counting softmax), one 1024 layer may work too. NN should have wide hidden layers, at least in the bottom.
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-22T01:11:25.683Z
Comments.Text: Does your deep layer (exp 3) have skip connections / residual connections? If not, then the gradient is likely vanishing, and early layers aren't learning anything much useful
Comments.UserDisplayName: user3667125
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-22T01:13:27.787Z
Comments.Text: I would use SGD over Adam, as my experience with Adam on small network and small dataset is that it finds a worse optimum than SGD. Additionally, what batch size are you using for neural network? It affects the optimum as well. Smaller gives better generalization, so try something small like 8.
Comments.UserDisplayName: user3667125
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-22T01:16:51.003Z
Comments.Text: Would be helpful to know if exp1/exp2 is overfitting the data more so than exp3. What is both the training accuracy and test accuracy for exp1, exp2, and exp3?
Comments.UserDisplayName: user3667125
Tags: <neural-networks><deep-learning><overfitting><multilayer-perceptrons>
Title: Unable to overfit using MLP
UserDisplayName: Ashwin Kannan
ThreadText: # Unable to overfit using MLP
  
  (Asked by: username_0 on 2018-05-06)
  
  I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples).
  
  With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.
  
  Now moving on to MLP results,
  
  **Exp 1:**
  
  * *Network Architecture:* [67 40 5]
  * *Optimizer:* Adam
  * *Learning Rate:* exponential decay of base learning rate
  * *Validation Accuracy:* ~45%
  * *Observation:* Both training accuracy and validation accuracy stops improving.
  
  **Exp 2:**
  Repeated **Exp 1** with batchnorm layer
  
  * *Validation Accuracy:* ~50%
  * *Observation:* Got 5% increase in accuracy.
  
  **Exp 3:**
  
  To overfit, increased the depth of MLP. A deeper version of **Exp 1** network
  
  * *Network Architecture:* [67 40 40 40 40 40 40 5]
  * *Optimizer:* Adam
  * *Learning Rate:* exponential decay of base learning rate
  * *Validation Accuracy:* ~55%
  
  Thoughts on what might be happening?
  
  **Tags:** neural-networks, deep-learning, overfitting, multilayer-perceptrons
  
  > **Comments:**
  > **username_1**: Such results are not unexpected. 67 dim vector is small, multiclass SVM would likely produce better results then NN. NN start to shine with dimensionality >=1000. If you actually want overfit, just for fun, you probably need bigger network with 128 to 512 layer size, at least 3 layers (not counting softmax), one 1024 layer may work too. NN should have wide hidden layers, at least in the bottom.  
  > **username_2**: Does your deep layer (exp 3) have skip connections / residual connections? If not, then the gradient is likely vanishing, and early layers aren't learning anything much useful  
  > **username_2**: I would use SGD over Adam, as my experience with Adam on small network and small dataset is that it finds a worse optimum than SGD. Additionally, what batch size are you using for neural network? It affects the optimum as well. Smaller gives better generalization, so try something small like 8.  
  > **username_2**: Would be helpful to know if exp1/exp2 is overfitting the data more so than exp3. What is both the training accuracy and test accuracy for exp1, exp2, and exp3?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2021-04-21. Score: 0
  
  For the multiclass SVM, there will be an ensembling effect since you are learning 5\*4=20 1vs1 classifiers. It could be an interesting experiment to try the same thing with simple neural networks. Also, since you are standardizing the inputs you could try tanh activations on the first layer after the input. I presume you are using softmax on the output layer.
  
  
  ---
  
  ### Answer by username_4 on 2018-05-06. Score: 0
  
  I guess you are using linear activation functions, maybe you are not initializing your weights, or you are regularizing your model enough.
  
  Initialize weights with [glorot](http://www.jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf?hc_location=ufi), insert dropout layers in between, use Relu as your activation function, stop the training process based on Early Stopping. and just experiment with one hidden layer.
  
  side note: if you use Adam, don't mess with the learning rate.  
  in SGD optimizer you could use decay because there is a single learning rate for all weight updates and the learning rate does not change during training.  
  In Adam, the learning rate is maintained for each network weight (parameter) and separately adopted as learning unfolds.
  
  > **Comments:**
  > **username_0**: I'm using pytorch to model MLP. I'm assuming weight initialization is handled by pytorch. Also, I'm using ReLU and regularization is 0 by default in pytorch.  Also, what sense does it make to use dropout layer when I'm not able to overfit on train data?  I'll look into the learning rate aspect you mentioned.  
  
  ---
  
  