Answers.Body: <p>It's hard to know for sure what's gone wrong, but here are some possibilities:</p>
  
  <ol>
  <li><p>The problem is difficult. The COCO  <a href="https://arxiv.org/pdf/1405.0312.pdf" rel="nofollow noreferrer">paper</a> reports that a typical object covers just 4-6% of the image. A randomly initialized model is therefore likely to do extremely poorly, with an expected precision of between 4 and 6% for detecting the object of interest in a frame. You also have 90 classes in your configuration file. It's not clear whether the model has access to the correct label, but if it's also inferring the class, we'd expect initial accuracy somewhere around 0.06%. That's actually about the precision of your starting model.</p></li>
  <li><p>You're training on mini-batches of size 32. It's not clear to me from your config whether the lower axis of that graph is iterations or epochs. If it's the former, then your model will have seen only 1.5 million examples during the entire training period. For a problem this hard, that's not nearly enough. Indeed, COCO contains only 328k unique examples, so the model would have seen each of them just 5 times (corresponding to 5 epochs). If you've done 500,000 epochs though, then that ought to be enough.</p></li>
  <li><p>It's possible that the hyper-parameters are not well set. I'm not an expert at training CNNs, but deep networks are notoriously finicky. I have difficulty reading your config file, but your learning rate looks reasonable to me. It also appears to undergo exponential decay over time though, and it's not clear to me that you want to be doing that, or whether the schedule you're using for the decay makes sense. This might be worth reviewing.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T17:57:23.750Z
Answers.Text: Thank you John Doucette for providing me such a detailed answer. Anyway:
  1. Right now I got 8% accuracy
  2. The lower axis represents a training step(=one gradient update).
  3. I can't change hyperparameters. I can't change anything into the configuration. This is because I'm working with the new Google Vision Kit and this is the only configuration compatible.
Answers.UserDisplayName: Giacomo Bartoli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T17:57:39.847Z
Answers.Text: PS: the drop in accuracy happened because I tried to set a learning rate 10 times higher than the original. As soon as I realized that accuracy was decreasing I re-set the learning rate as the beginning and then the accuracy started to increase again. This is the result (almost 700k iterations): https://ibb.co/fKFwbU
Answers.UserDisplayName: Giacomo Bartoli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T18:30:27.690Z
Answers.Text: Checking the TF documentation I can say that, for instance, if iteration 50 means we're processing the 50th batch.
Answers.UserDisplayName: Giacomo Bartoli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T21:15:09.943Z
Answers.Text: @GiacomoBartoli Ah, if 50 iterations = 50 batches, you'll need to train a while longer! Probably on the order of 100 times longer would be my guess.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T21:43:36.363Z
Answers.Text: 100 times longer means something like 100 weeks = 23 months of training. The real question is: when someone trains a network from scratch, which accuracy level is expected to obtain before move on and apply transfer learning?
  What developers do, typically, is to start from a pre-trained model and then train only the last layers on a custom dataset (this is what I mean for transfer learning).
  How much accuracy does the pre-trained model have?
Answers.UserDisplayName: Giacomo Bartoli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T15:33:45.270Z
Answers.LastActivityDate: 2018-08-14T15:33:45.270Z
Answers.UserDisplayName: John Doucette
Body: <p>I am using Tensorflow Object Detection API for training a CNN from scratch on COCO dataset. I need to use this <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/embedded_ssd_mobilenet_v1_coco.config" rel="nofollow noreferrer">specific configuration</a>.
  There is no pre-trained model on COCO with that configuration and this is the reason why I am training from scratch.</p>
  
  <p>However, after 1 week of training and evaluating each checkpoint generated by the training phase this is how my learning phase appears on Tensorboard:</p>
  
  <p><a href="https://i.sstatic.net/0U89O.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0U89O.png" alt="Tensorboard eval"></a></p>
  
  <p>Thus, my questions are:</p>
  
  <ul>
  <li>does anyone know how many iterations approximately will be necessary? Right now I did more than 500'000 iterations.</li>
  <li>How can be possible that after 500'000 the evaluation is 0,8%? I would expected something like 60-70%.</li>
  <li>Why does there is a sudden drop after 500k iterations? I thought that the eval was supposed to converge to some limit. (this is what SGD should do)</li>
  <li>Is there any 'trick' to speed up the training phase? (ex: increasing the learning rate, etc).</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-14T01:27:12.517Z
Comments.Text: It might be useful to know a little more about your problem. What would the precision of guessing uniformly at random look like? How much data do you have?
  
  Note too, that SGD is not guaranteed to converge smoothly, because you might update repeatedly based on an unfortunate random sample. It converges only in expectation.
Comments.UserDisplayName: John Doucette
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-14T07:36:28.903Z
Comments.Text: I'm training on coco dataset, which is 18GB of labeled data. I've no idea how much the precision of guessing uniformly at random should look like.
Comments.UserDisplayName: Giacomo Bartoli
Tags: <convolutional-neural-networks><computer-vision><tensorflow><object-recognition>
Title: Training a CNN from scratch over COCO dataset
UserDisplayName: Giacomo Bartoli
ThreadText: # Training a CNN from scratch over COCO dataset
  
  (Asked by: username_0 on 2018-08-13)
  
  I am using Tensorflow Object Detection API for training a CNN from scratch on COCO dataset. I need to use this [specific configuration](https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/embedded_ssd_mobilenet_v1_coco.config).
  There is no pre-trained model on COCO with that configuration and this is the reason why I am training from scratch.
  
  However, after 1 week of training and evaluating each checkpoint generated by the training phase this is how my learning phase appears on Tensorboard:
  
  [![Tensorboard eval](https://i.sstatic.net/0U89O.png)](https://i.sstatic.net/0U89O.png)
  
  Thus, my questions are:
  
  * does anyone know how many iterations approximately will be necessary? Right now I did more than 500'000 iterations.
  * How can be possible that after 500'000 the evaluation is 0,8%? I would expected something like 60-70%.
  * Why does there is a sudden drop after 500k iterations? I thought that the eval was supposed to converge to some limit. (this is what SGD should do)
  * Is there any 'trick' to speed up the training phase? (ex: increasing the learning rate, etc).
  
  **Tags:** convolutional-neural-networks, computer-vision, tensorflow, object-recognition
  
  > **Comments:**
  > **username_1**: It might be useful to know a little more about your problem. What would the precision of guessing uniformly at random look like? How much data do you have?  Note too, that SGD is not guaranteed to converge smoothly, because you might update repeatedly based on an unfortunate random sample. It converges only in expectation.  
  > **username_0**: I'm training on coco dataset, which is 18GB of labeled data. I've no idea how much the precision of guessing uniformly at random should look like.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-14. Score: 1
  
  It's hard to know for sure what's gone wrong, but here are some possibilities:
  
  1. The problem is difficult. The COCO [paper](https://arxiv.org/pdf/1405.0312.pdf) reports that a typical object covers just 4-6% of the image. A randomly initialized model is therefore likely to do extremely poorly, with an expected precision of between 4 and 6% for detecting the object of interest in a frame. You also have 90 classes in your configuration file. It's not clear whether the model has access to the correct label, but if it's also inferring the class, we'd expect initial accuracy somewhere around 0.06%. That's actually about the precision of your starting model.
  2. You're training on mini-batches of size 32. It's not clear to me from your config whether the lower axis of that graph is iterations or epochs. If it's the former, then your model will have seen only 1.5 million examples during the entire training period. For a problem this hard, that's not nearly enough. Indeed, COCO contains only 328k unique examples, so the model would have seen each of them just 5 times (corresponding to 5 epochs). If you've done 500,000 epochs though, then that ought to be enough.
  3. It's possible that the hyper-parameters are not well set. I'm not an expert at training CNNs, but deep networks are notoriously finicky. I have difficulty reading your config file, but your learning rate looks reasonable to me. It also appears to undergo exponential decay over time though, and it's not clear to me that you want to be doing that, or whether the schedule you're using for the decay makes sense. This might be worth reviewing.
  
  > **Comments:**
  > **username_0**: Thank you username_1 username_1 for providing me such a detailed answer. Anyway: 1. Right now I got 8% accuracy 2. The lower axis represents a training step(=one gradient update). 3. I can't change hyperparameters. I can't change anything into the configuration. This is because I'm working with the new Google Vision Kit and this is the only configuration compatible.  
  > **username_0**: PS: the drop in accuracy happened because I tried to set a learning rate 10 times higher than the original. As soon as I realized that accuracy was decreasing I re-set the learning rate as the beginning and then the accuracy started to increase again. This is the result (almost 700k iterations): https://ibb.co/fKFwbU  
  > **username_0**: Checking the TF documentation I can say that, for instance, if iteration 50 means we're processing the 50th batch.  
  > **username_1**: @username_0 Ah, if 50 iterations = 50 batches, you'll need to train a while longer! Probably on the order of 100 times longer would be my guess.  
  > **username_0**: 100 times longer means something like 100 weeks = 23 months of training. The real question is: when someone trains a network from scratch, which accuracy level is expected to obtain before move on and apply transfer learning? What developers do, typically, is to start from a pre-trained model and then train only the last layers on a custom dataset (this is what I mean for transfer learning). How much accuracy does the pre-trained model have?  
  
  ---
  
  