Answers.Body: <blockquote>
  <p>I'm sure the biases are initially initialized to zero but I don't know how the weights are handled.</p>
  </blockquote>
  <p>Looking at the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense" rel="nofollow noreferrer">Dense layer docs</a>: by default Dense layers biases are initialized with zeros (<code>bias_initializer='zeros'</code>) and weights are initialized with <a href="https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotUniform" rel="nofollow noreferrer">Glorot uniform</a> (<code>kernel_initializer='glorot_uniform'</code>).</p>
  <blockquote>
  <p>... &quot;unusual&quot; element to point here;  I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner.</p>
  </blockquote>
  <p>There's nothing unusual about the oscillations. Quite on the contrary - your curves are suspiciously <em>too</em> smooth.</p>
  <blockquote>
  <p>I've also noticed that the losses provided by history.history are different</p>
  </blockquote>
  <p>Yes, so this is a little gotcha in the <code>keras</code> implementation. For training loss, <code>keras</code> does a running average over the batches throughout an epoch, while for the validation loss it computes it after the epoch finishes. (<a href="https://github.com/keras-team/keras/issues/10426#issuecomment-397485072" rel="nofollow noreferrer">link</a>)</p>
  <p>That also explains why your validation loss starts slightly lower than the training one - it actually lags behind the training loss by <span class="math-container">$\sim1/2$</span>  of an epoch.</p>
  <p>The fact that both losses stay almost equal suggests that your model don't really rely on training data for prediction - it got saturated (&quot;dying ReLus&quot;) almost instantly.</p>
  <p>Couple of suggestions that I can make:</p>
  <ul>
  <li><p>I don't see you setting up a learning rate. Try making it smaller (like, <em>much</em> smaller). And see if there's any difference.</p>
  </li>
  <li><p>MSE is a nasty loss - especially at large values. Have you tried standardizing the target values?</p>
  </li>
  </ul>
  <blockquote>
  <p>Does the number of neurons, hence the number of weights influence the phenomena of &quot;dying ReLus&quot; ?</p>
  </blockquote>
  <p>Yes, the less neurons you have the higher the chance that all of the neurons will die out. You can get some intuition about it on <a href="https://playground.tensorflow.org/#activation=relu&amp;batchSize=10&amp;dataset=spiral&amp;regDataset=reg-plane&amp;learningRate=0.01&amp;regularizationRate=0&amp;noise=0&amp;networkShape=6,6,6,6&amp;seed=0.68973&amp;showTestData=false&amp;discretize=false&amp;percTrainData=50&amp;x=true&amp;y=true&amp;xTimesY=false&amp;xSquared=false&amp;ySquared=false&amp;cosX=false&amp;sinX=false&amp;cosY=false&amp;sinY=false&amp;collectStats=false&amp;problem=classification&amp;initZero=false&amp;hideText=false" rel="nofollow noreferrer">https://playground.tensorflow.org/</a> - following the link, I've got a relu network that trains reasonably well. Try increasing the learning rate and observe saturation in the neurons. Reduce the number of neurons and see how the whole net gets stuck.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-17T05:21:24.690Z
Answers.Text: thank you for your answer and sorry for my late reply. You're right what's suspicious is the smoothness of the curve, normally we must notice the oscillations we notice in the logs. I've tried both making the learning rate smaller ($0.0001$) and standardizing the target values and it seems it's working quite well this time. Both training and validation losses decrease over the epochs and when they stabilize there is some difference between the two values (about $0.1$).
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-17T05:25:56.753Z
Answers.Text: Can you tell me please how did you came up with the solutions ? I don't understand how decreasing the learning rate can help against the "dying ReLus". A big learning rate can make us miss the minima of the loss function and it can shift the weights in a direction that can cause our ReLus to die but it can also "drive them away" from that direction am I wrong ? And I don't see how MSE helps us avoid that issue as well. Thank you in advance.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-17T12:27:06.307Z
Answers.Text: @Daviiid So, I think the lesson for you is that you _always_ have to do hyperparameter tuning. That's something that is usually not stressed enough in guides and tutorials.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-17T12:32:49.850Z
Answers.Text: @Daviid Just playing around with various parameters on the playground site will give you some intuition on what various parameters do. One advise is - don't be afraid of adding more layers and neurons.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-17T12:36:26.787Z
Answers.Text: @Daviiid It is hard to give extended answers in the comments (that's not how this site is supposed to work anyway). Some of your questions merit separate posts. I'd also suggest that you formulate you questions by talking about your particular predicament, but in more general context. "How decreasing the learning rate can help against the dying ReLus" is a good start.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-19T06:56:57.550Z
Answers.Text: I see, I'll be sure to remember that advice everytime.Thank you again a lot for all the help and explanations !
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-14T15:00:21.907Z
Answers.LastActivityDate: 2021-04-14T15:00:21.907Z
Answers.UserDisplayName: Kostya
Body: <p>I'm trying to predict the continuous values of a variable <span class="math-container">$y$</span> using a Fully Connected Neural Network while providing it with data from a <span class="math-container">$(3300, 13)$</span> matrix <span class="math-container">$X$</span> where <span class="math-container">$X[i, :]=[0,...,1,...,0,x_{i}]$</span>. So the first <span class="math-container">$12$</span> elements of a data vector are all zeros except for one element which is equal to <span class="math-container">$1$</span> to denote  the belonging of this data to a category. I'd like to add that my <span class="math-container">$X$</span> data is normalized with regard to the <span class="math-container">$13$</span>-th column and that both <span class="math-container">$X$</span> and <span class="math-container">$y$</span> are shuffled in the same manner. Please find below my code for my model:</p>
  <pre><code>model = Sequential()
  model.add(Input(shape=(13,)))
  model.add(Dense(6, activation = 'relu'))
  model.add(Dense(2, activation = 'relu'))
  model.add(Dense(1))
  
  model.compile(loss = 'mean_squared_error',
                optimizer = 'adam',
                metrics = ['RootMeanSquaredError'])
  
  history = model.fit(X, y, validation_split = 0.1, epochs=64)
  </code></pre>
  <p>When trying to plot the learning curve using:</p>
  <pre><code>plt.plot(history.history['loss'])
  plt.plot(history.history['val_loss'])
  plt.title('model loss')
  plt.ylabel('rmse')
  plt.xlabel('epoch')
  plt.legend(['train', 'val'], loc='upper left')
  plt.show()
  </code></pre>
  <p>I get these curves:</p>
  <p><a href="https://i.sstatic.net/db4nI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/db4nI.png" alt="enter image description here" /></a></p>
  <p>There's already an &quot;unusual&quot; element to point here; I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner but we don't notice that on the training learning curve.  For example the last four values of the loss are: <span class="math-container">$2.5176$</span>, <span class="math-container">$3.4718$</span>, <span class="math-container">$3.0704$</span> and it settles down on <span class="math-container">$3.8177$</span>. I've also noticed that the losses provided by <code>history.history</code> are different than those shown during training, I suspect ones are computed before the epoch and ones after but I'm not sure.</p>
  <p>I've tried to predict on the <span class="math-container">$275$</span> first elements of the training data. Most of the predictions took the value <span class="math-container">$4.2138872e+00$</span> but there are other predictions who took lesser values. I've computed the maximum of the predictions on the whole training set and it is <span class="math-container">$4.2138872e+00$</span>.</p>
  <p>I've also tried to train on the whole training set without a validation set to see what'll happen. I've made sure to rerun the cells of the model so that it doesn't take the weights it already found. I've noticed the same behaviour for the loss during training, but this time there is no constant predicted value that comes up as a maximum limit for the predictions.</p>
  <p>I've already asked this question <a href="https://ai.stackexchange.com/questions/27282/why-my-fully-connected-neural-network-outputs-the-same-prediction">here</a> and a user suggested to me that I should ask this question separately while providing the whole code. I ran the same code I was running and that was giving me the same predictions for no matter for my input vectors.</p>
  <p>I think, as the user @Kostya that answered my previous question pointed out, what's happening here is called &quot;dying ReLus&quot;. It's the same code that I'm running over and over but gives different predictions and the only random parameters are the weights and the biases. I'm sure the biases are initially initialized to zero but I don't know how the weights are handled. I suppose they're randomly generated by a centered and reduced normal distribution.</p>
  <p>I have last came to this question: does the number of neurons, hence the number of weights influence the phenomena of &quot;dying ReLus&quot; ? I came to think that because if we had a large number of weights, their values are likely to fill that interval where the majority of the probability mass is concentrated. And since we have a small number of weights, we can get some &quot;outlier&quot; weights which lead to dyind ReLus.</p>
  
Tags: <regression><relu><weights-initialization><dense-layers><learning-curve>
Title: How to explain that a same DNN model have radically different behaviours with each new initialization and training?
UserDisplayName: Daviiid
ThreadText: # How to explain that a same DNN model have radically different behaviours with each new initialization and training?
  
  (Asked by: username_0 on 2021-04-14)
  
  I'm trying to predict the continuous values of a variable $y$ using a Fully Connected Neural Network while providing it with data from a $(3300, 13)$ matrix $X$ where $X[i, :]=[0,...,1,...,0,x\_{i}]$. So the first $12$ elements of a data vector are all zeros except for one element which is equal to $1$ to denote the belonging of this data to a category. I'd like to add that my $X$ data is normalized with regard to the $13$-th column and that both $X$ and $y$ are shuffled in the same manner. Please find below my code for my model:
  
  ```
  model = Sequential()
  model.add(Input(shape=(13,)))
  model.add(Dense(6, activation = 'relu'))
  model.add(Dense(2, activation = 'relu'))
  model.add(Dense(1))
  
  model.compile(loss = 'mean_squared_error',
                optimizer = 'adam',
                metrics = ['RootMeanSquaredError'])
  
  history = model.fit(X, y, validation_split = 0.1, epochs=64)
  
  ```
  
  When trying to plot the learning curve using:
  
  ```
  plt.plot(history.history['loss'])
  plt.plot(history.history['val_loss'])
  plt.title('model loss')
  plt.ylabel('rmse')
  plt.xlabel('epoch')
  plt.legend(['train', 'val'], loc='upper left')
  plt.show()
  
  ```
  
  I get these curves:
  
  [![enter image description here](https://i.sstatic.net/db4nI.png)](https://i.sstatic.net/db4nI.png)
  
  There's already an "unusual" element to point here; I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner but we don't notice that on the training learning curve. For example the last four values of the loss are: $2.5176$, $3.4718$, $3.0704$ and it settles down on $3.8177$. I've also noticed that the losses provided by `history.history` are different than those shown during training, I suspect ones are computed before the epoch and ones after but I'm not sure.
  
  I've tried to predict on the $275$ first elements of the training data. Most of the predictions took the value $4.2138872e+00$ but there are other predictions who took lesser values. I've computed the maximum of the predictions on the whole training set and it is $4.2138872e+00$.
  
  I've also tried to train on the whole training set without a validation set to see what'll happen. I've made sure to rerun the cells of the model so that it doesn't take the weights it already found. I've noticed the same behaviour for the loss during training, but this time there is no constant predicted value that comes up as a maximum limit for the predictions.
  
  I've already asked this question [here](https://ai.stackexchange.com/questions/27282/why-my-fully-connected-neural-network-outputs-the-same-prediction) and a user suggested to me that I should ask this question separately while providing the whole code. I ran the same code I was running and that was giving me the same predictions for no matter for my input vectors.
  
  I think, as the user @username_1 that answered my previous question pointed out, what's happening here is called "dying ReLus". It's the same code that I'm running over and over but gives different predictions and the only random parameters are the weights and the biases. I'm sure the biases are initially initialized to zero but I don't know how the weights are handled. I suppose they're randomly generated by a centered and reduced normal distribution.
  
  I have last came to this question: does the number of neurons, hence the number of weights influence the phenomena of "dying ReLus" ? I came to think that because if we had a large number of weights, their values are likely to fill that interval where the majority of the probability mass is concentrated. And since we have a small number of weights, we can get some "outlier" weights which lead to dyind ReLus.
  
  **Tags:** regression, relu, weights-initialization, dense-layers, learning-curve
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-14. Score: 1 | Accepted answer
  
  > I'm sure the biases are initially initialized to zero but I don't know how the weights are handled.
  
  Looking at the [Dense layer docs](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense): by default Dense layers biases are initialized with zeros (`bias_initializer='zeros'`) and weights are initialized with [Glorot uniform](https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotUniform) (`kernel_initializer='glorot_uniform'`).
  
  > ... "unusual" element to point here; I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner.
  
  There's nothing unusual about the oscillations. Quite on the contrary - your curves are suspiciously *too* smooth.
  
  > I've also noticed that the losses provided by history.history are different
  
  Yes, so this is a little gotcha in the `keras` implementation. For training loss, `keras` does a running average over the batches throughout an epoch, while for the validation loss it computes it after the epoch finishes. ([link](https://github.com/keras-team/keras/issues/10426#issuecomment-397485072))
  
  That also explains why your validation loss starts slightly lower than the training one - it actually lags behind the training loss by $\sim1/2$ of an epoch.
  
  The fact that both losses stay almost equal suggests that your model don't really rely on training data for prediction - it got saturated ("dying ReLus") almost instantly.
  
  Couple of suggestions that I can make:
  
  * I don't see you setting up a learning rate. Try making it smaller (like, *much* smaller). And see if there's any difference.
  * MSE is a nasty loss - especially at large values. Have you tried standardizing the target values?
  
  > Does the number of neurons, hence the number of weights influence the phenomena of "dying ReLus" ?
  
  Yes, the less neurons you have the higher the chance that all of the neurons will die out. You can get some intuition about it on [https://playground.tensorflow.org/](https://playground.tensorflow.org/#activation=relu&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.01&regularizationRate=0&noise=0&networkShape=6,6,6,6&seed=0.68973&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false) - following the link, I've got a relu network that trains reasonably well. Try increasing the learning rate and observe saturation in the neurons. Reduce the number of neurons and see how the whole net gets stuck.
  
  > **Comments:**
  > **username_0**: thank you for your answer and sorry for my late reply. You're right what's suspicious is the smoothness of the curve, normally we must notice the oscillations we notice in the logs. I've tried both making the learning rate smaller ($0.0001$) and standardizing the target values and it seems it's working quite well this time. Both training and validation losses decrease over the epochs and when they stabilize there is some difference between the two values (about $0.1$).  
  > **username_0**: Can you tell me please how did you came up with the solutions ? I don't understand how decreasing the learning rate can help against the "dying ReLus". A big learning rate can make us miss the minima of the loss function and it can shift the weights in a direction that can cause our ReLus to die but it can also "drive them away" from that direction am I wrong ? And I don't see how MSE helps us avoid that issue as well. Thank you in advance.  
  > **username_1**: @username_0 So, I think the lesson for you is that you _always_ have to do hyperparameter tuning. That's something that is usually not stressed enough in guides and tutorials.  
  > **username_1**: @Daviid Just playing around with various parameters on the playground site will give you some intuition on what various parameters do. One advise is - don't be afraid of adding more layers and neurons.  
  > **username_1**: @username_0 It is hard to give extended answers in the comments (that's not how this site is supposed to work anyway). Some of your questions merit separate posts. I'd also suggest that you formulate you questions by talking about your particular predicament, but in more general context. "How decreasing the learning rate can help against the dying ReLus" is a good start.  
  > **username_0**: I see, I'll be sure to remember that advice everytime.Thank you again a lot for all the help and explanations !  
  
  ---
  
  