Answers.Body: <blockquote>
    <p>Is there anything else I could do to improve accuracy for both training and testing? </p>
  </blockquote>
  
  <p>Yes, of course, there are a lot of methods if you want to try to improve your accuracy, some that I can mention:</p>
  
  <ul>
  <li>Try to use a more complex model: ResNet, DenseNet, etc.</li>
  <li>Try to use other optimizers: Adam, Adadelta, etc.</li>
  <li>Tune your hyperparameters (e.g. change your learning rate, momentum, rescale factor, convolution size, number of feature maps, epochs, neurons, FC layers)</li>
  <li>Try to analyze your data, with ~75% and 4 categories, is that possible there is one category that difficult to classified?</li>
  </ul>
  
  <p>In essence, you have to do a lot of experiments with your model until you think "it is enough" (if you have a deadline). If you don't have a hard deadline, you can keep improving updating your ML model.</p>
  
  <blockquote>
    <p>How can I know if this is the best accuracy I can get?</p>
  </blockquote>
  
  <p>No, you can't until you compare it with other models/hyperparameters. If you do more experiments (some ways like the one I mentioned above) or compare with other people's experiments that using the same data, you'll find which one is the best. For an academic paper, for example, you need to compare at least 3 to 4 models that similar or experiment with hundreds of different hyperparameters combination.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T06:00:20.050Z
Answers.Text: thank you. currently i will be trying Resnet, also analyzing the missclassified data is a good point that i missed.  regarding the optimizers, i thought they were related to the type of the network task, like for classification task SGD is often used.
Answers.UserDisplayName: norahik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-25T01:38:06.183Z
Answers.LastActivityDate: 2019-03-25T01:38:06.183Z
Answers.UserDisplayName: malioboro
Answers.Body: <p>One option is not mentioned by <strong>malioboro</strong> is getting more data. Getting bigger dataset is almost always improve training results. If it's too hard to obtain more labeled data you can use data augmentation on existing data - small random transformations while keeping the same label.</p>
  
  <p>For images most common methods of augmentations are (applying padding if needed):</p>
  
  <ul>
  <li>small random zooming in/out </li>
  <li>small random shifts of image</li>
  <li>adding random noise to image</li>
  <li>small random change in brightness, contrast, color balance and similar parameters</li>
  </ul>
  
  <p>There are more complex methods, but they are dependent on specific of dataset/goal of training</p>
  
  <p>Stopping: </p>
  
  <ul>
  <li>you should stop training if you see training error is not decreasing - your have reached (possibly local) minima.</li>
  <li>you should stop if testing or validation error start to increase - your method is overfitting (data augmentation can help in that case)</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-26T07:53:32.193Z
Answers.LastActivityDate: 2019-03-26T07:59:20.003Z
Answers.LastEditDate: 2019-03-26T07:59:20.003Z
Answers.UserDisplayName: mirror2image
Body: <p>I am training a modified VGG-16 to classify crowd density (empty, low, moderate, high). 2 dropout layers were added at the end on the network each one after one of the last 2 FC layers.</p>
  
  <p>network settings:</p>
  
  <ul>
  <li><p>training data contain 4381 images categorized under 4 categories (empty, low, moderate, high), 20% of the training data is set for validation. test data has 2589 images.</p></li>
  <li><p>training is done for 50 epochs.(training validation accuracy drops after
  50 epochs)</p></li>
  <li><p>lr=0.001, decay=0.0005, momentum=0.9</p></li>
  <li><p>loss= categorical_crossentropy</p></li>
  <li><p>augmentation for (training, validation and testing data): rescale=1./255, brightness_range=(0.2,0.9), horizontal_flip</p></li>
  </ul>
  
  <p>With the above-stated settings, I get the following results: </p>
  
  <ul>
  <li><p>training evaluation loss: 0.59, accuracy: 0.77</p></li>
  <li><p>testing accuracy 77.5 (correct predictions 2007 out of 2589)</p></li>
  </ul>
  
  <p>Regarding this, I have two concerns: </p>
  
  <ol>
  <li><p>Is there anything else I could do to improve accuracy for both training and testing? </p></li>
  <li><p>How can I know if this is the best accuracy I can get?</p></li>
  </ol>
  
Tags: <convolutional-neural-networks><classification><performance><vgg>
Title: How do I improve accuracy and know when to stop training?
UserDisplayName: norahik
ThreadText: # How do I improve accuracy and know when to stop training?
  
  (Asked by: username_0 on 2019-03-24)
  
  I am training a modified VGG-16 to classify crowd density (empty, low, moderate, high). 2 dropout layers were added at the end on the network each one after one of the last 2 FC layers.
  
  network settings:
  
  * training data contain 4381 images categorized under 4 categories (empty, low, moderate, high), 20% of the training data is set for validation. test data has 2589 images.
  * training is done for 50 epochs.(training validation accuracy drops after
    50 epochs)
  * lr=0.001, decay=0.0005, momentum=0.9
  * loss= categorical\_crossentropy
  * augmentation for (training, validation and testing data): rescale=1./255, brightness\_range=(0.2,0.9), horizontal\_flip
  
  With the above-stated settings, I get the following results:
  
  * training evaluation loss: 0.59, accuracy: 0.77
  * testing accuracy 77.5 (correct predictions 2007 out of 2589)
  
  Regarding this, I have two concerns:
  
  1. Is there anything else I could do to improve accuracy for both training and testing?
  2. How can I know if this is the best accuracy I can get?
  
  **Tags:** convolutional-neural-networks, classification, performance, vgg
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-03-25. Score: 5
  
  > Is there anything else I could do to improve accuracy for both training and testing?
  
  Yes, of course, there are a lot of methods if you want to try to improve your accuracy, some that I can mention:
  
  * Try to use a more complex model: ResNet, DenseNet, etc.
  * Try to use other optimizers: Adam, Adadelta, etc.
  * Tune your hyperparameters (e.g. change your learning rate, momentum, rescale factor, convolution size, number of feature maps, epochs, neurons, FC layers)
  * Try to analyze your data, with ~75% and 4 categories, is that possible there is one category that difficult to classified?
  
  In essence, you have to do a lot of experiments with your model until you think "it is enough" (if you have a deadline). If you don't have a hard deadline, you can keep improving updating your ML model.
  
  > How can I know if this is the best accuracy I can get?
  
  No, you can't until you compare it with other models/hyperparameters. If you do more experiments (some ways like the one I mentioned above) or compare with other people's experiments that using the same data, you'll find which one is the best. For an academic paper, for example, you need to compare at least 3 to 4 models that similar or experiment with hundreds of different hyperparameters combination.
  
  > **Comments:**
  > **username_0**: thank you. currently i will be trying Resnet, also analyzing the missclassified data is a good point that i missed.  regarding the optimizers, i thought they were related to the type of the network task, like for classification task SGD is often used.  
  
  ---
  
  ### Answer by username_2 on 2019-03-26. Score: 3
  
  One option is not mentioned by **username_1** is getting more data. Getting bigger dataset is almost always improve training results. If it's too hard to obtain more labeled data you can use data augmentation on existing data - small random transformations while keeping the same label.
  
  For images most common methods of augmentations are (applying padding if needed):
  
  * small random zooming in/out
  * small random shifts of image
  * adding random noise to image
  * small random change in brightness, contrast, color balance and similar parameters
  
  There are more complex methods, but they are dependent on specific of dataset/goal of training
  
  Stopping:
  
  * you should stop training if you see training error is not decreasing - your have reached (possibly local) minima.
  * you should stop if testing or validation error start to increase - your method is overfitting (data augmentation can help in that case)
  
  
  ---
  
  