Answers.Body: <blockquote>
  <p>two hidden layers each comprising two neurons</p>
  </blockquote>
  <p>From your description it looks like that you only have 6 parameters for your inner layer (2x2 weight matrix + 2 biases). The whole network should be easy to interpret: you've got two 13-dimensional weight vectors <span class="math-container">$\vec{w}_1,\vec{w}_2$</span> that are dot-multiplied with the inputs, plus two biases <span class="math-container">$b$</span> and activation <span class="math-container">$\sigma$</span>:</p>
  <p><span class="math-container">$$ l_1 = \sigma\left(\vec{w}_1\vec{x} + b_1\right)$$</span>
  <span class="math-container">$$ l_2 = \sigma\left(\vec{w}_2\vec{x} + b_2\right)$$</span></p>
  <p>Then these two values are multiplied by 2x2 matrix + biases, then activation and linear combination.</p>
  <p>I'd look at how <span class="math-container">$l_1$</span> and <span class="math-container">$l_2$</span> are distributed. The fact that the outputs don't change is most likely due to the first layer getting saturated somehow. Look at the 13th dimension of <span class="math-container">$\vec{w}_i$</span> - it is likely to be large compared to other dimensions.</p>
  <p>First thing I'd try - standardizing your input 13th dimension, so it is distributed closer to <span class="math-container">$[0,1]$</span> ( or <span class="math-container">$[-1,1]$</span> ) range.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T03:54:12.277Z
Answers.Text: Thank you for your answer. For $w_{1}$ the largest value is at the 4th dimension then the 13th one. For $w_{2}$ it is the same but we can't say that the value of 13th dimension is really greater than the others, sometimes it's just the double. I get this issue while I have standardized my 13th dimension input sorry for not precising it in my post, I'll edit it. Do you have any other idea for solving this issue ? I'd like to add that I get null outputs from the beginning. I've taken the initialized weights and computed the outputs of the first layer. I got $0$s after applying the ReLu.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T03:57:48.060Z
Answers.Text: By the way can I ask you please how is the loss decreasing with each epochs while the outputs from the first layer are from the beginning of the training null ? Since the biases are initialized to zero too, it means that all outputs are zero. No weights would be updated am I wrong ?
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T12:42:14.090Z
Answers.Text: @Daviiid (For future: I'd suggest you avoid using the word "null" when talking bout zero values.) It seems that you are dealing with the "dying ReLU problem". Since you have only two neurons both of them likely get dead. In this case, the biases in 2nd layer and the output layer's linear coefficients would still able to train and converge to some average values, not accounting for your inputs.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T13:58:11.377Z
Answers.Text: you're right I shouldn't use "null" when talking about zero values. I don't understand quite well how the other layers' coefficients will train. From the first layer we get a $[l_{1} l_{2}]$, then if $W_{2}$ denotes the weights matrix for the second layer and $b_{2}$ the corresponding bias vector we get $W_{2}*[l_{1} l_{2}] + b_{2}$ but since the biases are initialized to zero and $[l_{1} l_{2}]=[0 0]$ we get zeros outputs through the second layer as well am I wrong ?
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T14:49:40.640Z
Answers.Text: @Daviiid  The outputs of the second are going into the final layer, which, as I understand, is another linear function. So, you'll have two more non-zero initialized weights there. (If I understood your description correctly)
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T16:00:51.040Z
Answers.Text: (I'd like to apologize first because I don't know how to fix the way your name isn't displayed after the @) The second layer contains 2 neurons, if we denote $W_{3}$ the matrix of weights for the final layer, $b_{3}$ the bias of the corresponding layer and $s_{1}, s_{2}$ the outputs of the second layer, we have for the final layer $W_{3}*[s_{1} s_{2}]+b_{3}$ but since $s_{1}, s_{2}$ are zeros we still get no update. Well I think so but I may be wrong.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T17:29:39.450Z
Answers.Text: @Daviiid You are right - if all the biases are 0, then you have to have 0 at the output. Are you sure your biases are 0 at the initialization? I'm afraid I'd have to look at your code to figure it out. I suggest you make a question focused on that.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-14T00:58:26.167Z
Answers.Text: Thank you for the advice, I shall take it and make a question focused on it right now.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-12T09:38:57.477Z
Answers.LastActivityDate: 2021-04-12T09:38:57.477Z
Answers.UserDisplayName: Kostya
Body: <p>I have a relatively small data set comprised of <span class="math-container">$3300$</span> data points where each data point is a <span class="math-container">$13$</span> dimensional vector where the <span class="math-container">$12$</span> first dimensions depict a &quot;category&quot; by taking the form of <span class="math-container">$[0,...,1,...,0]$</span> where <span class="math-container">$1$</span> is in the <span class="math-container">$i-th$</span> position for the <span class="math-container">$i-th$</span> category and the last dimension is an observation of a continuous variable, so typically one data point would be <span class="math-container">$[1,...,0,70.05]$</span>.
  I'm not trying to have something extremely accurate so I went with a Fully Connected Network with two hidden layers each comprising two neurons, the activation functions are ReLus, one neuron at the output layer because I'm trying to predict one value, and I didn't put any activation function for it. The optimizer is ADAM, the loss is the MSE while the metric is the RMSE.</p>
  <p>I get this learning curve below:
  <a href="https://i.sstatic.net/XIrPY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XIrPY.png" alt="Learning curve" /></a>
  Eventhough at the beginning the validation loss is lesser than the training loss (which I don't understand), I think at the end it show no sign of overfitting.</p>
  <p>What I don't understand is why my Neural Network predicting the same value as long as the <span class="math-container">$13-th$</span> dimension takes values greater than <span class="math-container">$5$</span> and that value is <span class="math-container">$0.9747201$</span>. If the <span class="math-container">$13-th$</span> dimension takes for example <span class="math-container">$4.9$</span> then the prediction would be <span class="math-container">$1.0005863$</span>. I thought that it has something to do with the ReLu but even when I switched to Sigmoid, I have this &quot;saturation&quot; effect. The value is different but I still get the same value when I pass a certain threshold.</p>
  <p>EDIT: I'd also like to add that I get this issue even with normalizing the 13th dimension (substracting the mean and dividing by the standard deviation).</p>
  <p>I'd like to add that all the values in my training and validation set are at least greater than <span class="math-container">$50$</span> if that may help.</p>
  
Tags: <neural-networks><activation-functions><dense-layers>
Title: Why my Fully Connected Neural Network outputs the same prediction?
UserDisplayName: Daviiid
ThreadText: # Why my Fully Connected Neural Network outputs the same prediction?
  
  (Asked by: username_0 on 2021-04-12)
  
  I have a relatively small data set comprised of $3300$ data points where each data point is a $13$ dimensional vector where the $12$ first dimensions depict a "category" by taking the form of $[0,...,1,...,0]$ where $1$ is in the $i-th$ position for the $i-th$ category and the last dimension is an observation of a continuous variable, so typically one data point would be $[1,...,0,70.05]$.
  I'm not trying to have something extremely accurate so I went with a Fully Connected Network with two hidden layers each comprising two neurons, the activation functions are ReLus, one neuron at the output layer because I'm trying to predict one value, and I didn't put any activation function for it. The optimizer is ADAM, the loss is the MSE while the metric is the RMSE.
  
  I get this learning curve below:
  [![Learning curve](https://i.sstatic.net/XIrPY.png)](https://i.sstatic.net/XIrPY.png)
  Eventhough at the beginning the validation loss is lesser than the training loss (which I don't understand), I think at the end it show no sign of overfitting.
  
  What I don't understand is why my Neural Network predicting the same value as long as the $13-th$ dimension takes values greater than $5$ and that value is $0.9747201$. If the $13-th$ dimension takes for example $4.9$ then the prediction would be $1.0005863$. I thought that it has something to do with the ReLu but even when I switched to Sigmoid, I have this "saturation" effect. The value is different but I still get the same value when I pass a certain threshold.
  
  EDIT: I'd also like to add that I get this issue even with normalizing the 13th dimension (substracting the mean and dividing by the standard deviation).
  
  I'd like to add that all the values in my training and validation set are at least greater than $50$ if that may help.
  
  **Tags:** neural-networks, activation-functions, dense-layers
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-12. Score: 1 | Accepted answer
  
  > two hidden layers each comprising two neurons
  
  From your description it looks like that you only have 6 parameters for your inner layer (2x2 weight matrix + 2 biases). The whole network should be easy to interpret: you've got two 13-dimensional weight vectors $\vec{w}\_1,\vec{w}\_2$ that are dot-multiplied with the inputs, plus two biases $b$ and activation $\sigma$:
  
  $$ l\_1 = \sigma\left(\vec{w}\_1\vec{x} + b\_1\right)$$
  $$ l\_2 = \sigma\left(\vec{w}\_2\vec{x} + b\_2\right)$$
  
  Then these two values are multiplied by 2x2 matrix + biases, then activation and linear combination.
  
  I'd look at how $l\_1$ and $l\_2$ are distributed. The fact that the outputs don't change is most likely due to the first layer getting saturated somehow. Look at the 13th dimension of $\vec{w}\_i$ - it is likely to be large compared to other dimensions.
  
  First thing I'd try - standardizing your input 13th dimension, so it is distributed closer to $[0,1]$ ( or $[-1,1]$ ) range.
  
  > **Comments:**
  > **username_0**: Thank you for your answer. For $w_{1}$ the largest value is at the 4th dimension then the 13th one. For $w_{2}$ it is the same but we can't say that the value of 13th dimension is really greater than the others, sometimes it's just the double. I get this issue while I have standardized my 13th dimension input sorry for not precising it in my post, I'll edit it. Do you have any other idea for solving this issue ? I'd like to add that I get null outputs from the beginning. I've taken the initialized weights and computed the outputs of the first layer. I got $0$s after applying the ReLu.  
  > **username_0**: By the way can I ask you please how is the loss decreasing with each epochs while the outputs from the first layer are from the beginning of the training null ? Since the biases are initialized to zero too, it means that all outputs are zero. No weights would be updated am I wrong ?  
  > **username_1**: @username_0 (For future: I'd suggest you avoid using the word "null" when talking bout zero values.) It seems that you are dealing with the "dying ReLU problem". Since you have only two neurons both of them likely get dead. In this case, the biases in 2nd layer and the output layer's linear coefficients would still able to train and converge to some average values, not accounting for your inputs.  
  > **username_0**: you're right I shouldn't use "null" when talking about zero values. I don't understand quite well how the other layers' coefficients will train. From the first layer we get a $[l_{1} l_{2}]$, then if $W_{2}$ denotes the weights matrix for the second layer and $b_{2}$ the corresponding bias vector we get $W_{2}*[l_{1} l_{2}] + b_{2}$ but since the biases are initialized to zero and $[l_{1} l_{2}]=[0 0]$ we get zeros outputs through the second layer as well am I wrong ?  
  > **username_1**: @username_0  The outputs of the second are going into the final layer, which, as I understand, is another linear function. So, you'll have two more non-zero initialized weights there. (If I understood your description correctly)  
  > **username_0**: (I'd like to apologize first because I don't know how to fix the way your name isn't displayed after the @) The second layer contains 2 neurons, if we denote $W_{3}$ the matrix of weights for the final layer, $b_{3}$ the bias of the corresponding layer and $s_{1}, s_{2}$ the outputs of the second layer, we have for the final layer $W_{3}*[s_{1} s_{2}]+b_{3}$ but since $s_{1}, s_{2}$ are zeros we still get no update. Well I think so but I may be wrong.  
  > **username_1**: @username_0 You are right - if all the biases are 0, then you have to have 0 at the output. Are you sure your biases are 0 at the initialization? I'm afraid I'd have to look at your code to figure it out. I suggest you make a question focused on that.  
  > **username_0**: Thank you for the advice, I shall take it and make a question focused on it right now.  
  
  ---
  
  