Answers.Body: <p>In the first example, Sigmoid activation cannot go below 0.  Either modify your encoding or switch to a different activation (tanh for example can go to -1).</p>
  <p>On the second issue, it is customary to use softmax activation on the outputs here which ensures all outputs sum to 1 and it can vaguely be interpreted as the network's estimated probability distribution over class (1 being 100% surity, .5 being 50%, so on).  This is not strictly necessary so you might also be looking at underfitting.  But I'd start with softmax and go from there.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-26T06:14:38.683Z
Answers.Text: Ok, It worked, for the softmax It was not necessary since i was already taking the max output lane. Turns out i swapped Sigmoid for tanh (Remembering It was Sigmoid in range -1,1). For First problem I reached 100% accuracy with  expected values. For second problema It was definitely underfitting. Now i reach accuracy 81% with some noise, I consider It great improvement. Thank you
Answers.UserDisplayName: CoffeDeveloper
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-25T21:54:43.190Z
Answers.LastActivityDate: 2024-03-25T21:54:43.190Z
Answers.UserDisplayName: foreverska
Body: <p>I made a neural network in C#, I observe the loss goes down but never below a certain treshold.</p>
  <p><strong>This is XOR function error graph:</strong>
  (The graph is every 4 samples, so for all the 4 possible combinations of inputs there is one combination that performs better (MIN) and one that perform worse (MAX error)</p>
  <p><a href="https://i.sstatic.net/JYdpO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JYdpO.png" alt="enter image description here" /></a></p>
  <p><strong>NN architecture:</strong></p>
  <ul>
  <li>4 layers</li>
  <li>Number of neurons (2,6,6,1)</li>
  <li>First 3 layers ReLU activated (He initi), last layer Sigmoid (Glorot Init)</li>
  <li>Epochs 5000 (20.000 samples)</li>
  <li>Learning rate: 0.02</li>
  </ul>
  <p><strong>Encoding</strong></p>
  <ul>
  <li>One is encoded as 0.5</li>
  <li>Zero is encoded as -0.5</li>
  </ul>
  <p>The function is almost working, in the sense it outputs almost 0 instead of -0.5 and it outputs almost 0.5 in place of 0.5.. So if I decode the input with the following</p>
  <p><strong>Decoding</strong></p>
  <ul>
  <li>One if output is &gt; 0.25</li>
  <li>Zer if output is &lt; 0.25</li>
  </ul>
  <p>it has 100% accuracy.. however since it was trained with -0.5 as encoded zero actually it is not outputting what I expect.</p>
  <hr />
  <p><strong>The second example is from MNIST:</strong></p>
  <p><a href="https://i.sstatic.net/CXNXN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CXNXN.png" alt="enter image description here" /></a></p>
  <p><strong>NN architecture:</strong></p>
  <ul>
  <li>Number of layers 3</li>
  <li>Number of neurons (784,128,10)</li>
  <li>All activations Sigmoid with Glorot initialization</li>
  <li>Learning rate 0.005</li>
  <li>Epochs 1 (the graph refers to 40% of the first epoch)</li>
  </ul>
  <p><strong>Encoding:</strong></p>
  <ul>
  <li>784 pixels encoded with grey values mapped from (0,255) to (0,1)</li>
  <li>10 channels in output where the output is set to 1 for corresponding digit.</li>
  </ul>
  <p>Even though the error rapidly decrease at the beginning, it gets stuck, and the accuracy stuck aroun 11% (slightly better than guessing the correct digit)</p>
  <hr />
  <p>So what are the possible causes for error staying above a certain treshold?</p>
  <p>Code: <a href="https://github.com/Darelbi/NeuraSharp/tree/main/NeuraSharp/NeuraSharp.BuiltIn" rel="nofollow noreferrer">https://github.com/Darelbi/NeuraSharp/tree/main/NeuraSharp/NeuraSharp.BuiltIn</a></p>
  
Tags: <neural-networks><loss>
Title: Loss goes down but never below a certain treshold
UserDisplayName: CoffeDeveloper
ThreadText: # Loss goes down but never below a certain treshold
  
  (Asked by: username_0 on 2024-03-25)
  
  I made a neural network in C#, I observe the loss goes down but never below a certain treshold.
  
  **This is XOR function error graph:**
  (The graph is every 4 samples, so for all the 4 possible combinations of inputs there is one combination that performs better (MIN) and one that perform worse (MAX error)
  
  [![enter image description here](https://i.sstatic.net/JYdpO.png)](https://i.sstatic.net/JYdpO.png)
  
  **NN architecture:**
  
  * 4 layers
  * Number of neurons (2,6,6,1)
  * First 3 layers ReLU activated (He initi), last layer Sigmoid (Glorot Init)
  * Epochs 5000 (20.000 samples)
  * Learning rate: 0.02
  
  **Encoding**
  
  * One is encoded as 0.5
  * Zero is encoded as -0.5
  
  The function is almost working, in the sense it outputs almost 0 instead of -0.5 and it outputs almost 0.5 in place of 0.5.. So if I decode the input with the following
  
  **Decoding**
  
  * One if output is > 0.25
  * Zer if output is < 0.25
  
  it has 100% accuracy.. however since it was trained with -0.5 as encoded zero actually it is not outputting what I expect.
  
  ---
  
  **The second example is from MNIST:**
  
  [![enter image description here](https://i.sstatic.net/CXNXN.png)](https://i.sstatic.net/CXNXN.png)
  
  **NN architecture:**
  
  * Number of layers 3
  * Number of neurons (784,128,10)
  * All activations Sigmoid with Glorot initialization
  * Learning rate 0.005
  * Epochs 1 (the graph refers to 40% of the first epoch)
  
  **Encoding:**
  
  * 784 pixels encoded with grey values mapped from (0,255) to (0,1)
  * 10 channels in output where the output is set to 1 for corresponding digit.
  
  Even though the error rapidly decrease at the beginning, it gets stuck, and the accuracy stuck aroun 11% (slightly better than guessing the correct digit)
  
  ---
  
  So what are the possible causes for error staying above a certain treshold?
  
  Code: <https://github.com/Darelbi/NeuraSharp/tree/main/NeuraSharp/NeuraSharp.BuiltIn>
  
  **Tags:** neural-networks, loss
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-25. Score: 2 | Accepted answer
  
  In the first example, Sigmoid activation cannot go below 0. Either modify your encoding or switch to a different activation (tanh for example can go to -1).
  
  On the second issue, it is customary to use softmax activation on the outputs here which ensures all outputs sum to 1 and it can vaguely be interpreted as the network's estimated probability distribution over class (1 being 100% surity, .5 being 50%, so on). This is not strictly necessary so you might also be looking at underfitting. But I'd start with softmax and go from there.
  
  > **Comments:**
  > **username_0**: Ok, It worked, for the softmax It was not necessary since i was already taking the max output lane. Turns out i swapped Sigmoid for tanh (Remembering It was Sigmoid in range -1,1). For First problem I reached 100% accuracy with  expected values. For second problema It was definitely underfitting. Now i reach accuracy 81% with some noise, I consider It great improvement. Thank you  
  
  ---
  
  