<p>In the first example, Sigmoid activation cannot go below 0.  Either modify your encoding or switch to a different activation (tanh for example can go to -1).</p>
<p>On the second issue, it is customary to use softmax activation on the outputs here which ensures all outputs sum to 1 and it can vaguely be interpreted as the network's estimated probability distribution over class (1 being 100% surity, .5 being 50%, so on).  This is not strictly necessary so you might also be looking at underfitting.  But I'd start with softmax and go from there.</p>

CC BY-SA 4.0
2024-03-26T06:14:38.683Z
Ok, It worked, for the softmax It was not necessary since i was already taking the max output lane. Turns out i swapped Sigmoid for tanh (Remembering It was Sigmoid in range -1,1). For First problem I reached 100% accuracy with  expected values. For second problema It was definitely underfitting. Now i reach accuracy 81% with some noise, I consider It great improvement. Thank you
CoffeDeveloper
CC BY-SA 4.0
2024-03-25T21:54:43.190Z
2024-03-25T21:54:43.190Z
foreverska
<p>I made a neural network in C#, I observe the loss goes down but never below a certain treshold.</p>
<p><strong>This is XOR function error graph:</strong>
(The graph is every 4 samples, so for all the 4 possible combinations of inputs there is one combination that performs better (MIN) and one that perform worse (MAX error)</p>
<p><a href="https://i.sstatic.net/JYdpO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JYdpO.png" alt="enter image description here" /></a></p>
<p><strong>NN architecture:</strong></p>
<ul>
<li>4 layers</li>
<li>Number of neurons (2,6,6,1)</li>
<li>First 3 layers ReLU activated (He initi), last layer Sigmoid (Glorot Init)</li>
<li>Epochs 5000 (20.000 samples)</li>
<li>Learning rate: 0.02</li>
</ul>
<p><strong>Encoding</strong></p>
<ul>
<li>One is encoded as 0.5</li>
<li>Zero is encoded as -0.5</li>
</ul>
<p>The function is almost working, in the sense it outputs almost 0 instead of -0.5 and it outputs almost 0.5 in place of 0.5.. So if I decode the input with the following</p>
<p><strong>Decoding</strong></p>
<ul>
<li>One if output is &gt; 0.25</li>
<li>Zer if output is &lt; 0.25</li>
</ul>
<p>it has 100% accuracy.. however since it was trained with -0.5 as encoded zero actually it is not outputting what I expect.</p>
<hr />
<p><strong>The second example is from MNIST:</strong></p>
<p><a href="https://i.sstatic.net/CXNXN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CXNXN.png" alt="enter image description here" /></a></p>
<p><strong>NN architecture:</strong></p>
<ul>
<li>Number of layers 3</li>
<li>Number of neurons (784,128,10)</li>
<li>All activations Sigmoid with Glorot initialization</li>
<li>Learning rate 0.005</li>
<li>Epochs 1 (the graph refers to 40% of the first epoch)</li>
</ul>
<p><strong>Encoding:</strong></p>
<ul>
<li>784 pixels encoded with grey values mapped from (0,255) to (0,1)</li>
<li>10 channels in output where the output is set to 1 for corresponding digit.</li>
</ul>
<p>Even though the error rapidly decrease at the beginning, it gets stuck, and the accuracy stuck aroun 11% (slightly better than guessing the correct digit)</p>
<hr />
<p>So what are the possible causes for error staying above a certain treshold?</p>
<p>Code: <a href="https://github.com/Darelbi/NeuraSharp/tree/main/NeuraSharp/NeuraSharp.BuiltIn" rel="nofollow noreferrer">https://github.com/Darelbi/NeuraSharp/tree/main/NeuraSharp/NeuraSharp.BuiltIn</a></p>

<neural-networks><loss>
Loss goes down but never below a certain treshold
CoffeDeveloper
# Loss goes down but never below a certain treshold

(Asked by: username_0 on 2024-03-25)

I made a neural network in C#, I observe the loss goes down but never below a certain treshold.

**This is XOR function error graph:**
(The graph is every 4 samples, so for all the 4 possible combinations of inputs there is one combination that performs better (MIN) and one that perform worse (MAX error)

[![enter image description here](https://i.sstatic.net/JYdpO.png)](https://i.sstatic.net/JYdpO.png)

**NN architecture:**

* 4 layers
* Number of neurons (2,6,6,1)
* First 3 layers ReLU activated (He initi), last layer Sigmoid (Glorot Init)
* Epochs 5000 (20.000 samples)
* Learning rate: 0.02

**Encoding**

* One is encoded as 0.5
* Zero is encoded as -0.5

The function is almost working, in the sense it outputs almost 0 instead of -0.5 and it outputs almost 0.5 in place of 0.5.. So if I decode the input with the following

**Decoding**

* One if output is > 0.25
* Zer if output is < 0.25

it has 100% accuracy.. however since it was trained with -0.5 as encoded zero actually it is not outputting what I expect.

---

**The second example is from MNIST:**

[![enter image description here](https://i.sstatic.net/CXNXN.png)](https://i.sstatic.net/CXNXN.png)

**NN architecture:**

* Number of layers 3
* Number of neurons (784,128,10)
* All activations Sigmoid with Glorot initialization
* Learning rate 0.005
* Epochs 1 (the graph refers to 40% of the first epoch)

**Encoding:**

* 784 pixels encoded with grey values mapped from (0,255) to (0,1)
* 10 channels in output where the output is set to 1 for corresponding digit.

Even though the error rapidly decrease at the beginning, it gets stuck, and the accuracy stuck aroun 11% (slightly better than guessing the correct digit)

---

So what are the possible causes for error staying above a certain treshold?

Code: <https://github.com/Darelbi/NeuraSharp/tree/main/NeuraSharp/NeuraSharp.BuiltIn>

**Tags:** neural-networks, loss


---

## 1 answers

### Answer by username_1 on 2024-03-25. Score: 2 | Accepted answer

In the first example, Sigmoid activation cannot go below 0. Either modify your encoding or switch to a different activation (tanh for example can go to -1).

On the second issue, it is customary to use softmax activation on the outputs here which ensures all outputs sum to 1 and it can vaguely be interpreted as the network's estimated probability distribution over class (1 being 100% surity, .5 being 50%, so on). This is not strictly necessary so you might also be looking at underfitting. But I'd start with softmax and go from there.

> **Comments:**
> **username_0**: Ok, It worked, for the softmax It was not necessary since i was already taking the max output lane. Turns out i swapped Sigmoid for tanh (Remembering It was Sigmoid in range -1,1). For First problem I reached 100% accuracy with  expected values. For second problema It was definitely underfitting. Now i reach accuracy 81% with some noise, I consider It great improvement. Thank you  

---

