Answers.Body: <p>When training our neural network, you need to scale your dataset in order to avoid slowing down the learning or prevent effective learning.
  Try normalizing your output.
  This <a href="https://machinelearningmastery.com/how-to-scale-data-for-long-short-term-memory-networks-in-python/" rel="nofollow noreferrer">Tutorial </a>might help</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T01:56:01.540Z
Answers.Text: @AggrajGupta See me answer. You should _also_ scale your data, but that's probably not the root of your problem.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-16T12:41:08.467Z
Answers.LastActivityDate: 2019-11-16T12:41:08.467Z
Answers.UserDisplayName: SUMAN
Answers.Body: <p>Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer. </p>
  
  <p>The sigmoid has the property that its derivative (S*(1-S)) will be extremely small when the activation function's value is close to 0 or close to 1. In fact, the largest it can be is about 0.25.</p>
  
  <p>The backpropigation algorithm, which is used to train a neural network, will propagate an error signal backwards. At each layer, the error signal will be multiplied by, among other things, the derivative of the activation function.</p>
  
  <p>It is therefore the case that by the 4th layer your signal is <em>at most</em> <span class="math-container">$0.25^4 = \frac{1}{256}$</span> the size that it was at the start of the network. In fact, it is likely much smaller than this. With a smaller signal, your learning rates at the bottom of the nextwork will effectively be much smaller than the learning rates at the top, which will make it very difficult to pick a learning rate that is effective overall.</p>
  
  <p>This problem is known as <a href="https://adventuresinmachinelearning.com/vanishing-gradient-problem-tensorflow/" rel="nofollow noreferrer">the vanishing gradient</a>. </p>
  
  <p>To fix this, if you want to use a deep architecture, consider using an activation function that does not suffer from a vanishing gradient. The <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">Rectified Linear</a> activation function, used in so-called "ReLU" units, is a non-linear activation that does not have a vanishing gradient. It is common to use ReLUs for the earlier layers in a network, and a sigmoid at the output layer, if you need outputs to be bounded between 0 and 1.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T01:55:29.953Z
Answers.LastActivityDate: 2019-11-19T02:13:14.170Z
Answers.LastEditDate: 2019-11-19T02:13:14.170Z
Answers.UserDisplayName: John Doucette
Body: <p>I have a large set of data points describing mappings of binary vectors to real-valued outputs. I am using TensorFlow, and would like to train a model to predict these relationships. I used four hidden layers with 500 neurons in each layer, and sigmoidal activation functions in each layer.</p>
  
  <p>The network appears to be unable to learn, and has high loss even on the training data. What might cause this to happen? Is there something wrong with the design of my network?</p>
  
Tags: <neural-networks><deep-learning><tensorflow><activation-functions><hidden-layers>
Title: What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?
UserDisplayName: Aggraj Gupta
ThreadText: # What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?
  
  (Asked by: username_0 on 2019-11-16)
  
  I have a large set of data points describing mappings of binary vectors to real-valued outputs. I am using TensorFlow, and would like to train a model to predict these relationships. I used four hidden layers with 500 neurons in each layer, and sigmoidal activation functions in each layer.
  
  The network appears to be unable to learn, and has high loss even on the training data. What might cause this to happen? Is there something wrong with the design of my network?
  
  **Tags:** neural-networks, deep-learning, tensorflow, activation-functions, hidden-layers
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-11-16. Score: 0 | Accepted answer
  
  When training our neural network, you need to scale your dataset in order to avoid slowing down the learning or prevent effective learning.
  Try normalizing your output.
  This [Tutorial](https://machinelearningmastery.com/how-to-scale-data-for-long-short-term-memory-networks-in-python/) might help
  
  > **Comments:**
  > **username_2**: @username_0 See me answer. You should _also_ scale your data, but that's probably not the root of your problem.  
  
  ---
  
  ### Answer by username_2 on 2019-11-19. Score: 3
  
  Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer.
  
  The sigmoid has the property that its derivative (S\*(1-S)) will be extremely small when the activation function's value is close to 0 or close to 1. In fact, the largest it can be is about 0.25.
  
  The backpropigation algorithm, which is used to train a neural network, will propagate an error signal backwards. At each layer, the error signal will be multiplied by, among other things, the derivative of the activation function.
  
  It is therefore the case that by the 4th layer your signal is *at most* $0.25^4 = \frac{1}{256}$ the size that it was at the start of the network. In fact, it is likely much smaller than this. With a smaller signal, your learning rates at the bottom of the nextwork will effectively be much smaller than the learning rates at the top, which will make it very difficult to pick a learning rate that is effective overall.
  
  This problem is known as [the vanishing gradient](https://adventuresinmachinelearning.com/vanishing-gradient-problem-tensorflow/).
  
  To fix this, if you want to use a deep architecture, consider using an activation function that does not suffer from a vanishing gradient. The [Rectified Linear](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) activation function, used in so-called "ReLU" units, is a non-linear activation that does not have a vanishing gradient. It is common to use ReLUs for the earlier layers in a network, and a sigmoid at the output layer, if you need outputs to be bounded between 0 and 1.
  
  
  ---
  
  