Answers.Body: <p>It's the loss function.</p>
  
  <p>I was using squared sum error, which I didn't think would have as a negative effect as it does, and I had to come to the explanation in my own time. Here's why:</p>
  
  <p>From the perspective of the loss function, 999 times out of 1000, the output should be 0, so there will be an inherent massive bias towards 0 for all the output nodes. But this only occurs if the output nodes are <em>actually</em> trained when their desired outputs are 0, which is what happens in the case of the squared sum/mean error. However, in the case of cross-entropy loss, which is explained excellently <a href="https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation">here</a>, you can see that the only node that receives gradients is the node that should be trained towards 1. This removes the massive bias towards 0, and punishes confident false positives, making it perfect for a classification problem.</p>
  
  <p>As to how you would achieve something like this for regression I do not know, but at least this solves the issue for classification problems.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-23T05:12:34.483Z
Answers.LastActivityDate: 2019-07-23T05:12:34.483Z
Answers.UserDisplayName: Recessive
Body: <p>Consider AlexNet, which has 1000 output nodes, each of which classifies an image:</p>
  
  <p><a href="https://i.sstatic.net/mAVsh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mAVsh.png" alt="enter image description here"></a></p>
  
  <p>The problem I have been having with training a neural network of similar proportions, is that it does what any reasonable network would do: it finds the <em>easiest</em> way to reduce the error which happens to be setting all nodes to 0, as in the vast majority of the time, that's what they'll be. I don't understand how a network where 999 times out of 1000, the node's output is 0, could possibly learn to make that node 1.</p>
  
  <p>But obviously, it's possible, as AlexNet did very well in the 2012 ImageNet challenge. So I wanted to know, how would one train a neural network (specifically a CNN) when for the majority of the inputs the desired value for an output node is 0?</p>
  
Tags: <convolutional-neural-networks><training><backpropagation><alexnet>
Title: How is a neural network where the majority of inputs are 0 trained?
UserDisplayName: Recessive
ThreadText: # How is a neural network where the majority of inputs are 0 trained?
  
  (Asked by: username_0 on 2019-07-22)
  
  Consider AlexNet, which has 1000 output nodes, each of which classifies an image:
  
  [![enter image description here](https://i.sstatic.net/mAVsh.png)](https://i.sstatic.net/mAVsh.png)
  
  The problem I have been having with training a neural network of similar proportions, is that it does what any reasonable network would do: it finds the *easiest* way to reduce the error which happens to be setting all nodes to 0, as in the vast majority of the time, that's what they'll be. I don't understand how a network where 999 times out of 1000, the node's output is 0, could possibly learn to make that node 1.
  
  But obviously, it's possible, as AlexNet did very well in the 2012 ImageNet challenge. So I wanted to know, how would one train a neural network (specifically a CNN) when for the majority of the inputs the desired value for an output node is 0?
  
  **Tags:** convolutional-neural-networks, training, backpropagation, alexnet
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-07-23. Score: 1 | Accepted answer
  
  It's the loss function.
  
  I was using squared sum error, which I didn't think would have as a negative effect as it does, and I had to come to the explanation in my own time. Here's why:
  
  From the perspective of the loss function, 999 times out of 1000, the output should be 0, so there will be an inherent massive bias towards 0 for all the output nodes. But this only occurs if the output nodes are *actually* trained when their desired outputs are 0, which is what happens in the case of the squared sum/mean error. However, in the case of cross-entropy loss, which is explained excellently [here](https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation), you can see that the only node that receives gradients is the node that should be trained towards 1. This removes the massive bias towards 0, and punishes confident false positives, making it perfect for a classification problem.
  
  As to how you would achieve something like this for regression I do not know, but at least this solves the issue for classification problems.
  
  
  ---
  
  