Answers.Body: <p>The simple answer to your question is &quot;No&quot; with a caveat.</p>
  <p>The caveat is that there are signs that your network is never going to perform well.  For example, the epoch accuracy fails to improve or even consistently declines over the first several epochs, or the validation accuracy is flat or declining.  It could be that the validation loss starts high and just keeps increasing from the beginning.  These are all bad signs.</p>
  <p>Outside of this, however, it's very tough to know the model won't work well in the long run.  For example, we have a model we built for solving a set of CAPTCHAs.  The regression portions of that converged very quickly, but the portions that solved the rest of the CAPTCHA took something like 18 hours before they converged.  Honestly, we only ran it that long because it was the end of the day and the regression piece looked so promising; there was nothing in the training behavior of the CAPTCHA solver that looked like it would work (even though our intuition was that it should.)</p>
  <p>In the end, we have a 96%+ accuracy CAPTCHA solver that we likely would have killed if we had watched it train for more than 10 or 15 minutes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-21T14:14:24.250Z
Answers.LastActivityDate: 2021-10-21T14:14:24.250Z
Answers.UserDisplayName: David Hoelzer
Answers.Body: <p>Your description sounds like something similar to <a href="http://www.image-net.org/" rel="nofollow noreferrer">Imagenet dataset</a>. According to <a href="https://paperswithcode.com/sota/image-classification-on-imagenet" rel="nofollow noreferrer">this website</a>, the state-of-the-art top-1 accuracy just as high as 86%, not much higher than yours. There are plenty of methods to improve accuracy. I would suggest you to read the paper or github listed in <a href="https://paperswithcode.com/sota/image-classification-on-imagenet" rel="nofollow noreferrer">SOTA</a> to find ideas that best fits your situation. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-02T00:01:32.687Z
Answers.LastActivityDate: 2019-10-02T00:01:32.687Z
Answers.UserDisplayName: Terence Hsu
Answers.Body: <p>I can't comment but here are a few suggestions:</p>
  
  <ul>
  <li>Play with lr and lr finder</li>
  <li>Use a pretrained model</li>
  <li>Use architecture search or use something like efficientnet b6</li>
  <li>Use swish over relu</li>
  <li>Try different optimizers</li>
  <li>Try Bayesian Optimization</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-02T05:01:31.970Z
Answers.LastActivityDate: 2019-10-02T05:01:31.970Z
Answers.UserDisplayName: Dimitris Monroe
Answers.Body: <p>I would like you to try with following changes in the model. </p>
  
  <ul>
  <li>Introduce batch normalization layer in the model. </li>
  <li>Try with the batch size of 32 - 64</li>
  <li>Use different architecture, Like VGG, Resnet, etc..</li>
  </ul>
  
  <p>There is no full proof answer to this question but you can get best by trying some know strategies.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-06T14:47:56.413Z
Answers.Text: Actually I did all your say 1) BN is the network. 2) The batch size is 50, tested with VGG19 and the results are similar to mine, also I have test with Vision AutoML from google with similar results.
Answers.UserDisplayName: Mquinteiro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-06T13:35:41.060Z
Answers.LastActivityDate: 2018-09-06T13:35:41.060Z
Answers.UserDisplayName: Patel Sunil
Body: <p>I developed a CNN for image analysis. I've around 100K labeled images.  I'm getting a accuracy around 85% and a validation accuracy around 82%, so it looks like the model generalize better than fitting. So, I'm playing with different hyper-parameters: number of filters, number of layers, number of neurons in the dense layers, etc. </p>
  
  <p>For every test, I'm using all the training data, and it is very slow and time consuming.  </p>
  
  <p>Is there a way to have an early idea about if a model will perform better than another?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-06T13:15:22.957Z
Comments.Text: Hi, you don't specify the expected accuracy in your question. The difference that you have written between 85% for training and 82% for validation seems normal and implies that the model generalises well. What is your expected accuracy, and how do you know it is expected (is there another experiment using the exact same data that you are trying to match)?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-06T13:21:19.120Z
Comments.Text: @NeilSlater yes, I think that it generalizes ok, even if I try to overfit I don't get better acc that 87% or so. Trained human accuracy could be around 98%. I've tried to use transfer learning with similar or worst results... The labeled data is made by my, no another experiment is using the exact data.
Comments.UserDisplayName: Mquinteiro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-07T18:52:18.073Z
Comments.Text: So your expectation of 98% is based on human accuracy on the same task?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-07T22:42:17.963Z
Comments.Text: @NeilSlater yes it is, or something close to it, 95% could be enough, but I'm far, now playing manually with the number of convolution layers and filters. so tedious
Comments.UserDisplayName: Mquinteiro
Tags: <machine-learning><training><deep-neural-networks>
Title: Is there a way of pre-determining whether a CNN model will perform better than another?
UserDisplayName: Mquinteiro
ThreadText: # Is there a way of pre-determining whether a CNN model will perform better than another?
  
  (Asked by: username_0 on 2018-09-06)
  
  I developed a CNN for image analysis. I've around 100K labeled images. I'm getting a accuracy around 85% and a validation accuracy around 82%, so it looks like the model generalize better than fitting. So, I'm playing with different hyper-parameters: number of filters, number of layers, number of neurons in the dense layers, etc.
  
  For every test, I'm using all the training data, and it is very slow and time consuming.
  
  Is there a way to have an early idea about if a model will perform better than another?
  
  **Tags:** machine-learning, training, deep-neural-networks
  > **Comments:**
  > **username_1**: Hi, you don't specify the expected accuracy in your question. The difference that you have written between 85% for training and 82% for validation seems normal and implies that the model generalises well. What is your expected accuracy, and how do you know it is expected (is there another experiment using the exact same data that you are trying to match)?  
  > **username_0**: @NeilSlater yes, I think that it generalizes ok, even if I try to overfit I don't get better acc that 87% or so. Trained human accuracy could be around 98%. I've tried to use transfer learning with similar or worst results... The labeled data is made by my, no another experiment is using the exact data.  
  > **username_1**: So your expectation of 98% is based on human accuracy on the same task?  
  > **username_0**: @NeilSlater yes it is, or something close to it, 95% could be enough, but I'm far, now playing manually with the number of convolution layers and filters. so tedious  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2021-10-21. Score: 1
  
  The simple answer to your question is "No" with a caveat.
  
  The caveat is that there are signs that your network is never going to perform well. For example, the epoch accuracy fails to improve or even consistently declines over the first several epochs, or the validation accuracy is flat or declining. It could be that the validation loss starts high and just keeps increasing from the beginning. These are all bad signs.
  
  Outside of this, however, it's very tough to know the model won't work well in the long run. For example, we have a model we built for solving a set of CAPTCHAs. The regression portions of that converged very quickly, but the portions that solved the rest of the CAPTCHA took something like 18 hours before they converged. Honestly, we only ran it that long because it was the end of the day and the regression piece looked so promising; there was nothing in the training behavior of the CAPTCHA solver that looked like it would work (even though our intuition was that it should.)
  
  In the end, we have a 96%+ accuracy CAPTCHA solver that we likely would have killed if we had watched it train for more than 10 or 15 minutes.
  
  
  ---
  
  ### Answer by username_3 on 2019-10-02. Score: 0
  
  Your description sounds like something similar to [Imagenet dataset](http://www.image-net.org/). According to [this website](https://paperswithcode.com/sota/image-classification-on-imagenet), the state-of-the-art top-1 accuracy just as high as 86%, not much higher than yours. There are plenty of methods to improve accuracy. I would suggest you to read the paper or github listed in [SOTA](https://paperswithcode.com/sota/image-classification-on-imagenet) to find ideas that best fits your situation.
  
  
  ---
  
  ### Answer by username_4 on 2019-10-02. Score: 0
  
  I can't comment but here are a few suggestions:
  
  * Play with lr and lr finder
  * Use a pretrained model
  * Use architecture search or use something like efficientnet b6
  * Use swish over relu
  * Try different optimizers
  * Try Bayesian Optimization
  
  
  ---
  
  ### Answer by username_5 on 2018-09-06. Score: 0
  
  I would like you to try with following changes in the model.
  
  * Introduce batch normalization layer in the model.
  * Try with the batch size of 32 - 64
  * Use different architecture, Like VGG, Resnet, etc..
  
  There is no full proof answer to this question but you can get best by trying some know strategies.
  
  > **Comments:**
  > **username_0**: Actually I did all your say 1) BN is the network. 2) The batch size is 50, tested with VGG19 and the results are similar to mine, also I have test with Vision AutoML from google with similar results.  
  
  ---
  
  