Answers.Body: <p>You results show signs of overfitting at around epoch 40. In order to overcome this you can either simplify the model somewhat or increase regularization. You do not share what values you are using for dropout regularization so you can try increasing that. </p>
  
  <p>But to be honest, I am not sure if that is going to help. You are using dropout in a pure CNN architecture, which I do not see that often in recent models. When they exists, they are usually at the very end of the network, or after densely connected layers. There are claims that regular dropout does not work as intended for convolutional layers, so the idea of <a href="https://arxiv.org/pdf/1411.4280.pdf" rel="nofollow noreferrer">spatial dropout</a> is developed. </p>
  
  <p>But you are using them after pooling layers and such a design <a href="https://arxiv.org/pdf/1512.01400.pdf" rel="nofollow noreferrer">may not be</a> as bad as using them after convolutional layers. In any case, if you check the <a href="http://en.wikipedia.org/wiki/CIFAR-10" rel="nofollow noreferrer">top performers</a> on this dataset, you can see there are some who are trying to come up with new regularization techniques for CNN's, such as <a href="https://arxiv.org/abs/1705.07485" rel="nofollow noreferrer">shake-shake</a>, <a href="https://arxiv.org/abs/1802.02375" rel="nofollow noreferrer">shakedrop</a> and <a href="https://arxiv.org/abs/1708.04552" rel="nofollow noreferrer">cutout</a>. </p>
  
  <p>However, given you already have an overfitting model, removing regularization might not be the best idea. But if you replace the layers after (and including) the last convolution with some fully connected layers, you can use dropout regularization on them instead. This would remove the largest convolutional layer of the model and you can add dense layers without increasing model size.</p>
  
  <p>For image recognition tasks of any kind, easiest way to achieve high accuracy is via <strong>transfer learning</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T14:53:50.433Z
Answers.LastActivityDate: 2019-11-21T07:42:11.177Z
Answers.LastEditDate: 2019-11-21T07:42:11.177Z
Answers.UserDisplayName: serali
Body: <p>I have achieved around 85% accuracy using the following architecture:
  <a href="https://i.sstatic.net/97JWn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/97JWn.png" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/bgeBW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bgeBW.png" alt="enter image description here"></a></p>
  
  <p>I used a learning rate of 0.001 and trained the model over 125 epochs with a batch size of 64.
  Any suggestions would be much appreciated. Thanks in advance.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-19T05:50:47.253Z
Comments.Text: Try applying a grid search to all your hyper parameters (such as dropout and learning rate). A grid search is easy to do and can improve your accuracy a lot. If you really wanted to know the best architecture, search up NAS (Neural Architecture Search). Keep in mind this is significantly more complicated.
Comments.UserDisplayName: Recessive
Tags: <neural-networks><convolutional-neural-networks>
Title: What could I do to this CNN to achieve a higher accuracy on the cifar10 dataset?
UserDisplayName: FeedMeInformation
ThreadText: # What could I do to this CNN to achieve a higher accuracy on the cifar10 dataset?
  
  (Asked by: username_0 on 2019-11-19)
  
  I have achieved around 85% accuracy using the following architecture:
  [![enter image description here](https://i.sstatic.net/97JWn.png)](https://i.sstatic.net/97JWn.png)
  [![enter image description here](https://i.sstatic.net/bgeBW.png)](https://i.sstatic.net/bgeBW.png)
  
  I used a learning rate of 0.001 and trained the model over 125 epochs with a batch size of 64.
  Any suggestions would be much appreciated. Thanks in advance.
  
  **Tags:** neural-networks, convolutional-neural-networks
  > **Comments:**
  > **username_1**: Try applying a grid search to all your hyper parameters (such as dropout and learning rate). A grid search is easy to do and can improve your accuracy a lot. If you really wanted to know the best architecture, search up NAS (Neural Architecture Search). Keep in mind this is significantly more complicated.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-11-19. Score: 2
  
  You results show signs of overfitting at around epoch 40. In order to overcome this you can either simplify the model somewhat or increase regularization. You do not share what values you are using for dropout regularization so you can try increasing that.
  
  But to be honest, I am not sure if that is going to help. You are using dropout in a pure CNN architecture, which I do not see that often in recent models. When they exists, they are usually at the very end of the network, or after densely connected layers. There are claims that regular dropout does not work as intended for convolutional layers, so the idea of [spatial dropout](https://arxiv.org/pdf/1411.4280.pdf) is developed.
  
  But you are using them after pooling layers and such a design [may not be](https://arxiv.org/pdf/1512.01400.pdf) as bad as using them after convolutional layers. In any case, if you check the [top performers](http://en.wikipedia.org/wiki/CIFAR-10) on this dataset, you can see there are some who are trying to come up with new regularization techniques for CNN's, such as [shake-shake](https://arxiv.org/abs/1705.07485), [shakedrop](https://arxiv.org/abs/1802.02375) and [cutout](https://arxiv.org/abs/1708.04552).
  
  However, given you already have an overfitting model, removing regularization might not be the best idea. But if you replace the layers after (and including) the last convolution with some fully connected layers, you can use dropout regularization on them instead. This would remove the largest convolutional layer of the model and you can add dense layers without increasing model size.
  
  For image recognition tasks of any kind, easiest way to achieve high accuracy is via **transfer learning**.
  
  
  ---
  
  