Answers.Body: <p>The activation function you are using is not differentiable, because it has a step at every half-integer value of <span class="math-container">$x$</span>. This could be a problem for training.</p>
  <p>I would expect a better result using the identity as the activation function.</p>
  <p>Other possible causes could be too small number of epochs or too small batch size. Your chosen value <span class="math-container">$1$</span> means Stochastic Gradient Descent, whose convergence tends to be more chaotic.</p>
  <p>However, the weights you get aren't too far from the expected <span class="math-container">$w_1=w_2=1$</span>, and their average is very close to it, so it looks like the code is probably right.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-10T10:52:25.913Z
Answers.Text: But i am not using the derivative of the activation function to calculate the next weights. Do you mean in general when applying gradient descend or also in this  case ? Also I was using the identify function before but that was causing the weights to change a lot. For ex if Y = 13 and YBar = 12.6 it would keep learning for it to be accurately 13 which takes forever with floating numbers.
Answers.UserDisplayName: EEAH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-10T11:18:23.333Z
Answers.Text: Mmm... The step in the round function gets a big change for some small changes in the input, which I bet makes convergence less smooth. If it takes forever that looks like a larger learning rate could help.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-10T11:32:22.967Z
Answers.Text: And that flat region in the round function also has the effect of killing data points: those whose sum is not yet right but under the rounding operation it looks like correct. I think it has the effect of reducing the effective batch size.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T12:28:09.940Z
Answers.Text: can you clarify more what you mean by "those whose sum is not yet right but under the rounding operation it looks like correct" ?
Answers.UserDisplayName: EEAH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T12:28:55.823Z
Answers.Text: Note this network should only deal with adding integers not floating numbers
Answers.UserDisplayName: EEAH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T12:37:19.617Z
Answers.Text: Sorry it's clear. some benificial correction to the weights is in ignored when because these flat regions in the rounding function, which is similar to reducing the batch size
Answers.UserDisplayName: EEAH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-09T06:42:46.010Z
Answers.LastActivityDate: 2023-04-09T08:27:12.050Z
Answers.LastEditDate: 2023-04-09T08:27:12.050Z
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.Body: <p>I advise to change the cost function to a <strong>mean squared error</strong> (as commonly done in regression problems): <span class="math-container">$\mathcal L(y,\hat{y}) = \frac{1}{2B}\sum_{i=0}^{B-1} (y^{(i)}-\hat{y}^{(i)})^2$</span>; where you average the squared differences over the batch size <span class="math-container">$B$</span>. Also it's common practice to divide by <span class="math-container">$1/2$</span> to remove the factor of <span class="math-container">$2$</span> when taking the derivative of this.</p>
  <p>Moreover, since the range of the data can vary a lot, use some <strong>weight decay</strong> to reduce the magnitude of the weights and prevent over-fitting to a given range of values: this may also help to reach the optimal of <span class="math-container">$w_1=w_2 = 1$</span>.</p>
  <p>Finally, prefer <em>larger batch sizes</em> (e.g. 64, 128, ...) since the noise in the gradient reduces, and generate at least <code>20x</code> more data samples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-09T18:24:57.207Z
Answers.Text: Although the cost is not squared, the weight update expression corresponds to the MSE.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-10T11:02:13.363Z
Answers.Text: Valid point for the weight decay and larger data set/batch size. But why does the cost function have any effect ? I am not using gradient descend for the need to have a differentiable cost function and this is a simple learning process which is not complex to make use of the benefits of mse.
Answers.UserDisplayName: EEAH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-10T11:22:09.040Z
Answers.Text: You use MSE implicitly because it is the summation of your cost expression, the difference of y's.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-09T17:44:30.957Z
Answers.LastActivityDate: 2023-04-09T17:44:30.957Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>This might be unnecessary but to learn the basics of neural networks, I am trying to create a single perceptron neural network to solve the adding operation of 2 inputs <code>(x1 + x2 = ybar)</code></p>
  <p>The code is written in C# since the math should be very simple and it's a language I use usually more than python.</p>
  <p><strong>The problem is the network is not capable of learning the right weights for this simple problem so I am trying to understand why is that before checking another working example on the internet</strong></p>
  <p><strong>Note before explaining more about the problem, feel free to include any level of math or technical stuff (although might not be needed for this simple question).</strong></p>
  <p><a href="https://i.sstatic.net/VpnHl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VpnHl.png" alt="enter image description here" /></a></p>
  <p><code>Learns</code> is the time the backpropagation happens. So equal to the nb of batches.
  Activation function takes x and rounds it to the nearest integer.
  The cost function is <code>yBar - y</code>. The total cost for each batch is the sum of all the costs generated by the samples in that batch divided by the batch size (so the average).
  Backpropagation is in the form of <code>w[i+1] = w[i] - learningRate * cost</code>.
  The hope is that <code>w1</code> and <code>w2</code> should end up having the same value after backpropagation.</p>
  <p>To test the code I tried first on 1 input:</p>
  <p><a href="https://i.sstatic.net/9R4uN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9R4uN.png" alt="enter image description here" /></a></p>
  <p>It learns a combination of weights for that case to work.</p>
  <p>But then I tried generating a random training set of 500 rows and changing the batch size to <code>10</code> expecting it to generalize the problem and the weights to be in the form of <code>w1 ~= w2</code>. But it is failing to do so.
  Most of the predictions are away from the correct value by 30 or 40 with some exceptions that are far by a 100.</p>
  <p>I am not sure about how to think of the problem. This is supposed to be a linear problem so the activation function should not be the issue.
  I suspected the way the cost is calculated might be the issue, but there doesn't seem to be a more fitting cost function for this adding operation.
  Tried changing the learning rate and the initial weights with being pretty sure that these are not the problem, and it also didn't work.</p>
  <p>How to think of this problem ? What are the clues to look for in order to find the wrong parameter ? And how to fix it ?</p>
  
Tags: <neural-networks><machine-learning><backpropagation><single-layer-perceptron>
Title: Why this single layer perceptron for the add operation not learning the correct weights?
UserDisplayName: EEAH
ThreadText: # Why this single layer perceptron for the add operation not learning the correct weights?
  
  (Asked by: username_0 on 2023-04-08)
  
  This might be unnecessary but to learn the basics of neural networks, I am trying to create a single perceptron neural network to solve the adding operation of 2 inputs `(x1 + x2 = ybar)`
  
  The code is written in C# since the math should be very simple and it's a language I use usually more than python.
  
  **The problem is the network is not capable of learning the right weights for this simple problem so I am trying to understand why is that before checking another working example on the internet**
  
  **Note before explaining more about the problem, feel free to include any level of math or technical stuff (although might not be needed for this simple question).**
  
  [![enter image description here](https://i.sstatic.net/VpnHl.png)](https://i.sstatic.net/VpnHl.png)
  
  `Learns` is the time the backpropagation happens. So equal to the nb of batches.
  Activation function takes x and rounds it to the nearest integer.
  The cost function is `yBar - y`. The total cost for each batch is the sum of all the costs generated by the samples in that batch divided by the batch size (so the average).
  Backpropagation is in the form of `w[i+1] = w[i] - learningRate * cost`.
  The hope is that `w1` and `w2` should end up having the same value after backpropagation.
  
  To test the code I tried first on 1 input:
  
  [![enter image description here](https://i.sstatic.net/9R4uN.png)](https://i.sstatic.net/9R4uN.png)
  
  It learns a combination of weights for that case to work.
  
  But then I tried generating a random training set of 500 rows and changing the batch size to `10` expecting it to generalize the problem and the weights to be in the form of `w1 ~= w2`. But it is failing to do so.
  Most of the predictions are away from the correct value by 30 or 40 with some exceptions that are far by a 100.
  
  I am not sure about how to think of the problem. This is supposed to be a linear problem so the activation function should not be the issue.
  I suspected the way the cost is calculated might be the issue, but there doesn't seem to be a more fitting cost function for this adding operation.
  Tried changing the learning rate and the initial weights with being pretty sure that these are not the problem, and it also didn't work.
  
  How to think of this problem ? What are the clues to look for in order to find the wrong parameter ? And how to fix it ?
  
  **Tags:** neural-networks, machine-learning, backpropagation, single-layer-perceptron
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-04-09. Score: 3
  
  The activation function you are using is not differentiable, because it has a step at every half-integer value of $x$. This could be a problem for training.
  
  I would expect a better result using the identity as the activation function.
  
  Other possible causes could be too small number of epochs or too small batch size. Your chosen value $1$ means Stochastic Gradient Descent, whose convergence tends to be more chaotic.
  
  However, the weights you get aren't too far from the expected $w\_1=w\_2=1$, and their average is very close to it, so it looks like the code is probably right.
  
  > **Comments:**
  > **username_0**: But i am not using the derivative of the activation function to calculate the next weights. Do you mean in general when applying gradient descend or also in this  case ? Also I was using the identify function before but that was causing the weights to change a lot. For ex if Y = 13 and YBar = 12.6 it would keep learning for it to be accurately 13 which takes forever with floating numbers.  
  > **username_1**: Mmm... The step in the round function gets a big change for some small changes in the input, which I bet makes convergence less smooth. If it takes forever that looks like a larger learning rate could help.  
  > **username_1**: And that flat region in the round function also has the effect of killing data points: those whose sum is not yet right but under the rounding operation it looks like correct. I think it has the effect of reducing the effective batch size.  
  > **username_0**: can you clarify more what you mean by "those whose sum is not yet right but under the rounding operation it looks like correct" ?  
  > **username_0**: Note this network should only deal with adding integers not floating numbers  
  > **username_0**: Sorry it's clear. some benificial correction to the weights is in ignored when because these flat regions in the rounding function, which is similar to reducing the batch size  
  
  ---
  
  ### Answer by username_2 on 2023-04-09. Score: 1
  
  I advise to change the cost function to a **mean squared error** (as commonly done in regression problems): $\mathcal L(y,\hat{y}) = \frac{1}{2B}\sum\_{i=0}^{B-1} (y^{(i)}-\hat{y}^{(i)})^2$; where you average the squared differences over the batch size $B$. Also it's common practice to divide by $1/2$ to remove the factor of $2$ when taking the derivative of this.
  
  Moreover, since the range of the data can vary a lot, use some **weight decay** to reduce the magnitude of the weights and prevent over-fitting to a given range of values: this may also help to reach the optimal of $w\_1=w\_2 = 1$.
  
  Finally, prefer *larger batch sizes* (e.g. 64, 128, ...) since the noise in the gradient reduces, and generate at least `20x` more data samples.
  
  > **Comments:**
  > **username_1**: Although the cost is not squared, the weight update expression corresponds to the MSE.  
  > **username_0**: Valid point for the weight decay and larger data set/batch size. But why does the cost function have any effect ? I am not using gradient descend for the need to have a differentiable cost function and this is a simple learning process which is not complex to make use of the benefits of mse.  
  > **username_1**: You use MSE implicitly because it is the summation of your cost expression, the difference of y's.  
  
  ---
  
  