Answers.Body: <p>Ok, so the fact that you always give the same category means the network only has to output the same value for any given input. This is exceedingly easy to learn (all the weights always shift in the same directions) and this make this graph. </p>
  
  <p>Once you have more categories (try 10 for example), you'll see the graph will be closer to expected.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-05T18:10:29.560Z
Answers.Text: Thats make a lot of sense. But why loss converges faster if my batch is too small? With sizes lower than 32, the networks converges with just 10% of an epoch (aprox. 100 images)
Answers.UserDisplayName: 648trindade
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-05T18:23:13.757Z
Answers.Text: You have less variance when you have a smaller batch size, so the network ignores the noise even faster
Answers.UserDisplayName: BlueMoon93
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-05T17:55:48.067Z
Answers.LastActivityDate: 2017-12-05T17:55:48.067Z
Answers.UserDisplayName: BlueMoon93
Body: <p>I'm training two CNNs (AlexNet e GoogLeNet) in two differents DL libraries (Caffe e Tensorflow). The networks was implemented by dev teams of each libraries (<a href="https://github.com/BVLC/caffe/tree/master/models" rel="nofollow noreferrer">here</a> and <a href="https://github.com/tensorflow/models/tree/master/research/slim/nets" rel="nofollow noreferrer">here</a>)</p>
  
  <p>I reduced the original Imagenet dataset to 1024 images of 1 category -- but setted 1000 categories to classify on the networks.</p>
  
  <p>So I trained the CNNs, varying processing unit (CPU/GPU) and batches sizes, and I observed that the losses converges fastly to near zero (in mostly times before 1 epoch be completed), like in this graph (Alexnet on Tensorflow):</p>
  
  <p><a href="https://i.sstatic.net/GXeEK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GXeEK.png" alt=""></a></p>
  
  <p>In portuguese, 'Épocas' is epochs and 'Perda' is loss.</p>
  
  <p>The weight decays and initial learning rate are the same as used on models that I downloaded, I only changed the dataset and the batch sizes.</p>
  
  <p>Why my networks are converging this way, and not like <a href="https://github.com/BVLC/caffe/issues/3801#issuecomment-194654755" rel="nofollow noreferrer">this way</a>?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-05T10:31:06.900Z
Comments.Text: What do you mean, you settled 1000 categories?
Comments.UserDisplayName: BlueMoon93
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-05T17:47:26.513Z
Comments.Text: I specified to network/library 1000 different output categories, but in practice I just give 1 to the network. I dont know exactly how Tensorflow/Caffe works, but I think that this value (1000) is used in softmax layer
Comments.UserDisplayName: 648trindade
Tags: <deep-learning><convolutional-neural-networks><training><tensorflow>
Title: What fast loss convergence indicates on a CNN?
UserDisplayName: 648trindade
ThreadText: # What fast loss convergence indicates on a CNN?
  
  (Asked by: username_0 on 2017-12-05)
  
  I'm training two CNNs (AlexNet e GoogLeNet) in two differents DL libraries (Caffe e Tensorflow). The networks was implemented by dev teams of each libraries ([here](https://github.com/BVLC/caffe/tree/master/models) and [here](https://github.com/tensorflow/models/tree/master/research/slim/nets))
  
  I reduced the original Imagenet dataset to 1024 images of 1 category -- but setted 1000 categories to classify on the networks.
  
  So I trained the CNNs, varying processing unit (CPU/GPU) and batches sizes, and I observed that the losses converges fastly to near zero (in mostly times before 1 epoch be completed), like in this graph (Alexnet on Tensorflow):
  
  [![](https://i.sstatic.net/GXeEK.png)](https://i.sstatic.net/GXeEK.png)
  
  In portuguese, 'Épocas' is epochs and 'Perda' is loss.
  
  The weight decays and initial learning rate are the same as used on models that I downloaded, I only changed the dataset and the batch sizes.
  
  Why my networks are converging this way, and not like [this way](https://github.com/BVLC/caffe/issues/3801#issuecomment-194654755)?
  
  **Tags:** deep-learning, convolutional-neural-networks, training, tensorflow
  
  > **Comments:**
  > **username_1**: What do you mean, you settled 1000 categories?  
  > **username_0**: I specified to network/library 1000 different output categories, but in practice I just give 1 to the network. I dont know exactly how Tensorflow/Caffe works, but I think that this value (1000) is used in softmax layer  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-12-05. Score: 2 | Accepted answer
  
  Ok, so the fact that you always give the same category means the network only has to output the same value for any given input. This is exceedingly easy to learn (all the weights always shift in the same directions) and this make this graph.
  
  Once you have more categories (try 10 for example), you'll see the graph will be closer to expected.
  
  > **Comments:**
  > **username_0**: Thats make a lot of sense. But why loss converges faster if my batch is too small? With sizes lower than 32, the networks converges with just 10% of an epoch (aprox. 100 images)  
  > **username_1**: You have less variance when you have a smaller batch size, so the network ignores the noise even faster  
  
  ---
  
  