Answers.Body: <p>How are you initializing the weights of each layer? It turns out that weight initialization plays a surprisingly large part in the network's ability to learn, but these issues are abstracted away in libraries <code>torch</code>. I wonder if that could be an issue here. See <a href="https://paperswithcode.com/method/he-initialization" rel="nofollow noreferrer">this paper</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T16:48:57.267Z
Answers.Text: This was the issue. I found out 14 hours ago when heading home. Got myself thinking about the weights initialization. Some time along the way I changed the weight initialization for some technique I do not recall the name of. It probably had some bug or wasn't appropriate. Just initialized between -0.5 and 0.5. But why does the initialization have so much impact on learning?
Answers.UserDisplayName: Evry
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T17:30:46.680Z
Answers.Text: Initializing the weights between -0.5 and 0.5 works fine
Answers.UserDisplayName: Evry
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T17:48:22.323Z
Answers.Text: Poor initialization can cause exploding or vanishing gradients - it's possible in your case that your network simply didn't learn anything because your gradient norms were too small. The reason I'm guessing that is because your network predicted only one class, which is typically what happens with randomly initialized networks
Answers.UserDisplayName: Julian L
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T17:51:03.050Z
Answers.Text: I see. Good to know that. Thanks for your time.
Answers.UserDisplayName: Evry
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T17:53:33.210Z
Answers.Text: Of course - honestly I am not an expert on the dynamics of model initialization. But a good paper is the Kaiming one I linked (that was huge for getting models to learn consistently) as well as Xavier init.
Answers.UserDisplayName: Julian L
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T18:23:45.540Z
Answers.Text: That is a good recommendation, I'll take a deeper look at the paper, thanks
Answers.UserDisplayName: Evry
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-03T20:04:10.680Z
Answers.Text: So, after some time I changed the weights initialization for the one you suggested. You were right, it was the initialization. Thank you very much
Answers.UserDisplayName: Evry
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T16:20:16.570Z
Answers.LastActivityDate: 2023-11-23T16:20:16.570Z
Answers.UserDisplayName: Julian L
Body: <p>I'm coding an FNN in Rust using the <a href="https://crates.io/crates/nalgebra" rel="nofollow noreferrer">nalgebra crate</a>. I coded the backpropagation based on <a href="https://brilliant.org/wiki/backpropagation/#:%7E:text=criterion%20is%20met.-,The%20Backpropagation%20Algorithm,-Using%20the%20terms" rel="nofollow noreferrer">this article</a> from Brilliant (the link directly highlights the formulas' section I).</p>
  <h1>The issue</h1>
  <p>My network tends to overclassify one class. See logs of some iterations:</p>
  <pre><code>Pred: [0.4957141833444244, 0.5042858166555757] Exp: [1.0, 0.0]
   Pred: [0.1159099292782023, 0.8840900707217978] Exp: [0.0, 1.0]
   Pred: [0.49812391110550713, 0.5018760888944928] Exp: [0.0, 1.0]
   Pred: [0.785823559592914, 0.21417644040708617] Exp: [1.0, 0.0]
  (0) Loss: 1.7553771084566945  accuracy: 75%
   Pred: [0.921104553204082, 0.07889544679591799] Exp: [1.0, 0.0]
   Pred: [0.43669346100767914, 0.5633065389923209] Exp: [0.0, 1.0]
   Pred: [0.8390020708741205, 0.16099792912587946] Exp: [0.0, 1.0]
   Pred: [0.9358457093202687, 0.0641542906797313] Exp: [1.0, 0.0]
  (1) Loss: 2.5487814857735516  accuracy: 75%
   Pred: [0.9938891101436632, 0.0061108898563368074] Exp: [1.0, 0.0]
   Pred: [0.8117580010095468, 0.18824199899045324] Exp: [0.0, 1.0]
   Pred: [0.9637103688436267, 0.03628963115637338] Exp: [0.0, 1.0]
   Pred: [0.9836993037515837, 0.01630069624841633] Exp: [1.0, 0.0]
  (2) Loss: 5.008814788847665  accuracy: 50%
   Pred: [0.9995224418411457, 0.0004775581588542201] Exp: [1.0, 0.0]
   Pred: [0.9640882324687131, 0.03591176753128681] Exp: [0.0, 1.0]
   Pred: [0.9940401356203591, 0.005959864379640862] Exp: [0.0, 1.0]
   Pred: [0.9967080323177023, 0.0032919676822977125] Exp: [1.0, 0.0]
  (3) Loss: 8.453172874424054  accuracy: 50%
   Pred: [0.9999695301335619, 3.0469866438138036e-5] Exp: [1.0, 0.0]
   Pred: [0.9955011694391566, 0.004498830560843456] Exp: [0.0, 1.0]
   Pred: [0.9993378187078414, 0.0006621812921585512] Exp: [0.0, 1.0]
   Pred: [0.9995329830642827, 0.00046701693571723986] Exp: [1.0, 0.0]
  (4) Loss: 12.724406571994546  accuracy: 50%
   Pred: [0.9999986403638063, 1.3596361937582465e-6] Exp: [1.0, 0.0]
   Pred: [0.9996152538707804, 0.0003847461292195687] Exp: [0.0, 1.0]
   Pred: [0.9999496312441125, 5.036875588738215e-5] Exp: [0.0, 1.0]
   Pred: [0.9999544427172501, 4.5557282750020274e-5] Exp: [1.0, 0.0]
  (5) Loss: 17.759113261749448  accuracy: 50%
   Pred: [0.9999999567632185, 4.323678145397258e-8] Exp: [1.0, 0.0]
   Pred: [0.9999768122330864, 2.3187766913582424e-5] Exp: [0.0, 1.0]
   Pred: [0.9999973081629956, 2.6918370044323987e-6] Exp: [0.0, 1.0]
   Pred: [0.9999969459673318, 3.054032668285032e-6] Exp: [1.0, 0.0]
  (6) Loss: 23.497175499984603  accuracy: 50%
  </code></pre>
  <p>This also happens when I have more classes, using other datasets (like the MNIST hand-digits).</p>
  <h1>My implementation</h1>
  <p>Based on the Brilliant article I coded a matrix based implementation of the Backpropagation, i.e. all calculations are made matrix-wise. I do not see any problems with my implementation. I also coded the RMSProp which seems to be fine since when I remove it and only use SGD to train the networks the same issue happens.</p>
  <p><strong>If you want to take a look at the <a href="https://github.com/thiago-buarqque/NeuraRust/tree/stackoverflow-question" rel="nofollow noreferrer">whole project</a>, I created a branch, which is a copy of the current main branch, that I'll never delete for the sake of this question.</strong></p>
  <h1>What I tried</h1>
  <ol>
  <li>Checked every activation and loss function I have coded (see my <a href="https://github.com/thiago-buarqque/NeuraRust/tree/stackoverflow-question/src/functions" rel="nofollow noreferrer">functions module code</a>). I'm using sigmoid for the hidden layer and softmax for the output layer. But even if I use relu for the hidden layer the issue is the same.</li>
  <li>Applied the math by hand, it seems to be check (I'm not an expert at math at all).</li>
  <li>Remove the RMSProp, just used SGD, same issue the network doesn't learn.</li>
  <li>Use the Relu activation function for the hidden layer.</li>
  </ol>
  <h1>Question</h1>
  <ol>
  <li>In theory, what could cause this?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-16T15:27:35.297Z
Comments.Text: Hi @Evry and welcome to AI Stack Exchange! Thank you so much for detailing what you have tried. This is an excellent question, and we hope to see more of your questions on this site soon!
Comments.UserDisplayName: DeepQZero
Tags: <gradient-descent><feedforward-neural-networks><rust>
Title: A Feedforward Neural Network (FNN) implemented with RMSProp optimization is exhibiting a tendency to overclassify instances into one particular class
UserDisplayName: Evry
ThreadText: # A Feedforward Neural Network (FNN) implemented with RMSProp optimization is exhibiting a tendency to overclassify instances into one particular class
  
  (Asked by: username_0 on 2023-11-16)
  
  I'm coding an FNN in Rust using the nalgebra crate. I coded the backpropagation based on this article from Brilliant (the link directly highlights the formulas' section I).
  The issue
  My network tends to overclassify one class. See logs of some iterations:
  Pred: [0.4957141833444244, 0.5042858166555757] Exp: [1.0, 0.0]
   Pred: [0.1159099292782023, 0.8840900707217978] Exp: [0.0, 1.0]
   Pred: [0.49812391110550713, 0.5018760888944928] Exp: [0.0, 1.0]
   Pred: [0.785823559592914, 0.21417644040708617] Exp: [1.0, 0.0]
  (0) Loss: 1.7553771084566945  accuracy: 75%
   Pred: [0.921104553204082, 0.07889544679591799] Exp: [1.0, 0.0]
   Pred: [0.43669346100767914, 0.5633065389923209] Exp: [0.0, 1.0]
   Pred: [0.8390020708741205, 0.16099792912587946] Exp: [0.0, 1.0]
   Pred: [0.9358457093202687, 0.0641542906797313] Exp: [1.0, 0.0]
  (1) Loss: 2.5487814857735516  accuracy: 75%
   Pred: [0.9938891101436632, 0.0061108898563368074] Exp: [1.0, 0.0]
   Pred: [0.8117580010095468, 0.18824199899045324] Exp: [0.0, 1.0]
   Pred: [0.9637103688436267, 0.03628963115637338] Exp: [0.0, 1.0]
   Pred: [0.9836993037515837, 0.01630069624841633] Exp: [1.0, 0.0]
  (2) Loss: 5.008814788847665  accuracy: 50%
   Pred: [0.9995224418411457, 0.0004775581588542201] Exp: [1.0, 0.0]
   Pred: [0.9640882324687131, 0.03591176753128681] Exp: [0.0, 1.0]
   Pred: [0.9940401356203591, 0.005959864379640862] Exp: [0.0, 1.0]
   Pred: [0.9967080323177023, 0.0032919676822977125] Exp: [1.0, 0.0]
  (3) Loss: 8.453172874424054  accuracy: 50%
   Pred: [0.9999695301335619, 3.0469866438138036e-5] Exp: [1.0, 0.0]
   Pred: [0.9955011694391566, 0.004498830560843456] Exp: [0.0, 1.0]
   Pred: [0.9993378187078414, 0.0006621812921585512] Exp: [0.0, 1.0]
   Pred: [0.9995329830642827, 0.00046701693571723986] Exp: [1.0, 0.0]
  (4) Loss: 12.724406571994546  accuracy: 50%
   Pred: [0.9999986403638063, 1.3596361937582465e-6] Exp: [1.0, 0.0]
   Pred: [0.9996152538707804, 0.0003847461292195687] Exp: [0.0, 1.0]
   Pred: [0.9999496312441125, 5.036875588738215e-5] Exp: [0.0, 1.0]
   Pred: [0.9999544427172501, 4.5557282750020274e-5] Exp: [1.0, 0.0]
  (5) Loss: 17.759113261749448  accuracy: 50%
   Pred: [0.9999999567632185, 4.323678145397258e-8] Exp: [1.0, 0.0]
   Pred: [0.9999768122330864, 2.3187766913582424e-5] Exp: [0.0, 1.0]
   Pred: [0.9999973081629956, 2.6918370044323987e-6] Exp: [0.0, 1.0]
   Pred: [0.9999969459673318, 3.054032668285032e-6] Exp: [1.0, 0.0]
  (6) Loss: 23.497175499984603  accuracy: 50%
  
  This also happens when I have more classes, using other datasets (like the MNIST hand-digits).
  My implementation
  Based on the Brilliant article I coded a matrix based implementation of the Backpropagation, i.e. all calculations are made matrix-wise. I do not see any problems with my implementation. I also coded the RMSProp which seems to be fine since when I remove it and only use SGD to train the networks the same issue happens.
  If you want to take a look at the whole project, I created a branch, which is a copy of the current main branch, that I'll never delete for the sake of this question.
  What I tried
  
  Checked every activation and loss function I have coded (see my functions module code). I'm using sigmoid for the hidden layer and softmax for the output layer. But even if I use relu for the hidden layer the issue is the same.
  Applied the math by hand, it seems to be check (I'm not an expert at math at all).
  Remove the RMSProp, just used SGD, same issue the network doesn't learn.
  Use the Relu activation function for the hidden layer.
  
  Question
  
  In theory, what could cause this?
  
  **Tags:** gradient-descent, feedforward-neural-networks, rust
  > **Comments:**
  > **username_1**: Hi @Evry and welcome to AI Stack Exchange! Thank you so much for detailing what you have tried. This is an excellent question, and we hope to see more of your questions on this site soon!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-11-23. Score: 0 | Accepted answer
  
  How are you initializing the weights of each layer? It turns out that weight initialization plays a surprisingly large part in the network's ability to learn, but these issues are abstracted away in libraries `torch`. I wonder if that could be an issue here. See [this paper](https://paperswithcode.com/method/he-initialization)
  
  > **Comments:**
  > **username_0**: This was the issue. I found out 14 hours ago when heading home. Got myself thinking about the weights initialization. Some time along the way I changed the weight initialization for some technique I do not recall the name of. It probably had some bug or wasn't appropriate. Just initialized between -0.5 and 0.5. But why does the initialization have so much impact on learning?  
  > **username_0**: Initializing the weights between -0.5 and 0.5 works fine  
  > **username_2**: Poor initialization can cause exploding or vanishing gradients - it's possible in your case that your network simply didn't learn anything because your gradient norms were too small. The reason I'm guessing that is because your network predicted only one class, which is typically what happens with randomly initialized networks  
  > **username_0**: I see. Good to know that. Thanks for your time.  
  > **username_2**: Of course - honestly I am not an expert on the dynamics of model initialization. But a good paper is the Kaiming one I linked (that was huge for getting models to learn consistently) as well as Xavier init.  
  > **username_0**: That is a good recommendation, I'll take a deeper look at the paper, thanks  
  > **username_0**: So, after some time I changed the weights initialization for the one you suggested. You were right, it was the initialization. Thank you very much  
  
  ---
  
  