Answers.Body: <p>Nothing to do with it! Let me illustrate this concept using one of the simplest models in machine learning: Linear Regression. In linear regression, the goal is to find the coefficients of a hyperplane that best fits the given data, represented by the equation <span class="math-container">$Ax=b$</span>. Here, we have free variables that represent the slopes along each axis and an independent term for the height (that in neural nets is called the Bias). Now, to address your question: Do the coefficients need to add up to 1? The answer is no.</p>
  <p>You might be tempted to interpret the concept of weights with the concept of probabilities, <strong>but they are not the same!</strong> They are merely coefficients in a high-dimensional, nested function.</p>
  <p><a href="https://i.sstatic.net/x4JIY.png" rel="noreferrer"><img src="https://i.sstatic.net/x4JIY.png" alt="enter image description here" /></a></p>
  <p>Here's a 3D example of linear regression. The blue points represent the data, and the red plane is the best-fitting hyperplane according to the linear regression model.</p>
  <p>The equation of the fitted plane is approximately:</p>
  <p><span class="math-container">$y=2.79⋅x_1+2.01⋅x_2+4.11$</span></p>
  <p>As you can see, the coefficients are  <span class="math-container">$2.79$</span> for <span class="math-container">$x_1$</span> <span class="math-container">$2.01$</span> for <span class="math-container">$x_2$</span> , and the intercept is <span class="math-container">$4.11$</span>. If we sum these together, we get <span class="math-container">$2.79 + 2.01 + 4.11 = 8.91$</span>, which is clearly not equal to <span class="math-container">$1$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-06T09:09:22.360Z
Answers.Text: Couldn't you simply reparameterise the coefficients to [0.2817, 0.2026, 0.4157] and have them add up to 1 again?
Answers.UserDisplayName: Toivo Säwén
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-06T14:27:34.840Z
Answers.Text: I think the main point here is the term "weights" has nothing to do with the term "weights" which represents a percentage.
Answers.UserDisplayName: osbm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-06T20:36:36.713Z
Answers.Text: @ToivoSäwén If we parameterize the coefficients to the values you mentioned in order for them to add up to one, we will not end up with the same hyperplane. In the example I gave, we had the following linear equation:
  
  y = 2.79x₁ + 2.01x₂ + 4.11
  
  If we divide the entire equation by 8.91, we get:
  
  y' = 0.31x₁ + 0.225x₂ + 0.416
  
  This changes the y value to y'. Therefore, we would have to change our data "y" value in order for it to work. I believe you are confusing normalizing a vector with the property that all its components add up to one.
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T01:30:56.027Z
Answers.LastActivityDate: 2023-09-04T01:30:56.027Z
Answers.UserDisplayName: Cesar Ruiz
Answers.Body: <p>No, weights in that case do not constitute a formal probability distribution and <strong>do not need to be add up to 1</strong>. In fact, if they would — it could cause problems with underflow. Imagine a (nonsensical) neural network — <em>n</em> inputs, followed by 10 layers, each containing 10k parameters. This MLP would possess 100,000 parameters, excluding biases and the output layer. Given the novel architectures, it is not even a large number of parameters. Now, we normalize all the weights (layer or network-wise) so they form a valid probability distribution. It is easy to see, that the forward and backward passes could be problematic as the value of each parameter would be close to 0. Also, each gradient could be negligible small - a phenomenon that is observable now in NN due to various reasons <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem" rel="nofollow noreferrer">Vanishing Gradient Problem</a>.</p>
  <p><strong>How do we interpret the activity of the Neural Networks</strong>? It is a reasonable question — there is a field of research that is called an <a href="https://en.wikipedia.org/wiki/Explainable_artificial_intelligence" rel="nofollow noreferrer">Explainable Artificial Intelligence</a> and deals with the interpretation of many deep learning architectures. Given that deep learning basically provides for creating a sequence of nested functions that can approximate arbitrary difficult non-linear functions it is a non-trivial task. If the weights would sum up to 1 — it could greatly help the progress. Unfortunately, this is not the case ;)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T11:54:22.837Z
Answers.LastActivityDate: 2023-09-06T12:48:35.863Z
Answers.LastEditDate: 2023-09-06T12:48:35.863Z
Answers.UserDisplayName: Scolpe
Answers.Body: <blockquote>
  <p>do all the weights need to add up to 1.0 (i.e. 100%)?</p>
  </blockquote>
  <p>No.</p>
  <blockquote>
  <p>If not, why?</p>
  </blockquote>
  <p>Why should they?</p>
  <p>In the case of school grades, the justification might be that each component (midterm exam, homework etc.) is graded separately on a same scale and the final grade, which <strong>must</strong> lie on the same scale, is computed as the weighted average of these components.</p>
  <p>There is no such requirement regarding weights in neural networks.</p>
  <p>What you <em>might</em> want to enforce is that the outputs in the last layer sum to one, to obtain probabilities of class memberships (this, obviously, applies only to classification problems and not to regression). Softmax is typically used for that purpose.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T11:53:11.403Z
Answers.LastActivityDate: 2023-09-04T11:53:11.403Z
Answers.UserDisplayName: Igor F.
Answers.Body: <p>No, the weights do not need to add up to one. There isn't really a reason to do that. Weights as &quot;contributions&quot; may not be the best way to think about things here -- you're trying to learn a <em>function</em> that transforms the input values into your desired output values. Say you're learning a linear model and your data is 2D and drawn from a line with slope 2, then you'd want your learned weight to be 2.</p>
  <p>Additionally, say you have a feature that is <em>negatively</em> correlated with the output. That is, that a higher value in the input results in a lower outputs value. If all your weights are non-negative, you wouldn't be able to encode this at all!</p>
  <p>Finally, as the output would simply be a weighted average of the inputs, the <em>scale</em> of the outputs would be constrained by the scale of the inputs. Consider:
  <span class="math-container">$$f(\mathbf{x}) = \Sigma^d_{i=1} w_i x_i$$</span>
  with
  <span class="math-container">$$\Sigma^d_{i=1} w_i = 1$$</span></p>
  <p><span class="math-container">$f(\mathbf{x})$</span> will always be less than or equal to <span class="math-container">$\max(\{w_1, ..., w_d\})$</span>. For a simple linear model, you can definitely think of examples where the scale of the output is much larger than that of the input features (for example, number of bathrooms in a house versus its price). Although you can mitigate some of this by rescaling your data (as I will talk about shortly), there is really no reason to artificially constrain your model like this.</p>
  <p>A deep neural network can be thought of as stacks of these linear models, with the output of the previous layer being the input of the next layer. Although, these intuitions aren't at all perfectly transferable to deep neural networks, we would still be constraining the expressiveness of the model in a similar way: an increase in an input dimension can't result in a decrease in the output value and the scale of the output is constrained by the scale of the input.</p>
  <h2>Normalization in Neural Networks</h2>
  <p>You <em>do</em>, however, often need to perform normalization, but this normalization is done on the data and the outputs of hidden layers instead of the weights. If you add a weight penalty to your linear model for example, you want to make sure that each dimension is in around the same range, to not overly penalize features that tend to have larger weights. In neural networks, you also often normalize weight values for hidden layers, the most common variant is <em>Batch Normalization</em>.</p>
  <p>However, the normalization often used in these cases is a bit different from what you mentioned. Also known as <em>standardization</em>, this kind of normalization subtracts each sample by the mean (over a set of samples) and divides by the standard deviation (over that same set of samples).</p>
  <p>The exact reason why this is effective in neural networks is <a href="https://arxiv.org/abs/1805.11604" rel="nofollow noreferrer">an active area of research</a>. A common explanation is <a href="https://en.wikipedia.org/wiki/Batch_normalization" rel="nofollow noreferrer">batch normalization reduces the effects of internal covariant shift</a>, where changes in weights during training results in changes in the output distribution of a layer, which can harm performance as the next layer &quot;expects&quot; the original distribution. Centering and scaling the input to have zero mean and unit standard deviation reduces these changes. This, however, is disputed by <a href="https://arxiv.org/pdf/1805.11604.pdf" rel="nofollow noreferrer">this paper</a>, which argues that batch normalization helps by making the loss landscape smoother.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T01:28:20.853Z
Answers.LastActivityDate: 2023-09-04T10:50:11.677Z
Answers.LastEditDate: 2023-09-04T10:50:11.677Z
Answers.UserDisplayName: Alexander Wan
Body: <p>The idea that weights determine how much influence each input value from the current layer will have when calculating the input to the following layer reminds me of when my professors would say that our final grade for the course was broken down something like this:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Item</th>
  <th>Percent of Grade</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Midterm Exam 1</td>
  <td>20%</td>
  </tr>
  <tr>
  <td>Midterm Exam 2</td>
  <td>20%</td>
  </tr>
  <tr>
  <td>Final Exam</td>
  <td>50%</td>
  </tr>
  <tr>
  <td>Classwork</td>
  <td>5%</td>
  </tr>
  <tr>
  <td>Homework</td>
  <td>5%</td>
  </tr>
  <tr>
  <td><strong>Total</strong></td>
  <td><strong>100%</strong></td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>My question is then do all the weights need to add up to 1.0 (i.e. 100%)? If not, why?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-04T17:10:49.780Z
Comments.Text: Why "If not, why?" and not "If yes, why?"
Comments.UserDisplayName: Stef
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-04T19:33:21.243Z
Comments.Text: @Stef I think I just assumed both were implied when I was thinking of the question, didn't think much of it, but valid point
Comments.UserDisplayName: Garrett
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-04T19:45:19.077Z
Comments.Text: ;) You'll notice one of the answers begins with *"No, the weights do not need to add up to one. There isn't really a reason to do that."* and another with *"No. Why should they?"*. There are situations where imposing the constraint "must sum to 1" is useful, but the weights of a neural network is not such a situation. Since it's not needed, we don't impose it. Scolpe's answer gives one argument why imposing it would be a bad idea. One place where this constraint would be useful is in the outputs of a classification neural net, where the outputs do represent probabilities (but not the weights)
Comments.UserDisplayName: Stef
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-05T09:24:10.430Z
Comments.Text: I don't understand what the relationship between the grades and the weights you thought could possibly exist. It seems that something is missing from this question. Some people in the answers even mention probability distributions, which you don't mention. This seems like a random question. Maybe a question that has not been asked here, or maybe yes. Still, the motivation for this question is unclear to me. It's unclear where the doubt is. It's unclear why you're asking this question. Still, it makes us think why. So, why not challenge the status quo?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-05T11:11:31.823Z
Comments.Text: @nbro I didn't originally think of weights in this way. I'm a visual learner and trying to think of real-world things I could compare weights to to help me visualize them and this idea came up.
Comments.UserDisplayName: Garrett
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-06T08:45:51.283Z
Comments.Text: Simply, the answer is "no".
Comments.UserDisplayName: Amazon Dies In Darkness
Tags: <neural-networks><deep-learning><weights>
Title: Do neural network weights need to add up to one?
UserDisplayName: Garrett
ThreadText: # Do neural network weights need to add up to one?
  
  (Asked by: username_0 on 2023-09-04)
  
  The idea that weights determine how much influence each input value from the current layer will have when calculating the input to the following layer reminds me of when my professors would say that our final grade for the course was broken down something like this:
  
  | Item | Percent of Grade |
  | --- | --- |
  | Midterm Exam 1 | 20% |
  | Midterm Exam 2 | 20% |
  | Final Exam | 50% |
  | Classwork | 5% |
  | Homework | 5% |
  | **Total** | **100%** |
  
  My question is then do all the weights need to add up to 1.0 (i.e. 100%)? If not, why?
  
  **Tags:** neural-networks, deep-learning, weights
  > **Comments:**
  > **username_1**: Why "If not, why?" and not "If yes, why?"  
  > **username_0**: @Stef I think I just assumed both were implied when I was thinking of the question, didn't think much of it, but valid point  
  > **username_1**: ;) You'll notice one of the answers begins with *"No, the weights do not need to add up to one. There isn't really a reason to do that."* and another with *"No. Why should they?"*. There are situations where imposing the constraint "must sum to 1" is useful, but the weights of a neural network is not such a situation. Since it's not needed, we don't impose it. Scolpe's answer gives one argument why imposing it would be a bad idea. One place where this constraint would be useful is in the outputs of a classification neural net, where the outputs do represent probabilities (but not the weights)  
  > **username_2**: I don't understand what the relationship between the grades and the weights you thought could possibly exist. It seems that something is missing from this question. Some people in the answers even mention probability distributions, which you don't mention. This seems like a random question. Maybe a question that has not been asked here, or maybe yes. Still, the motivation for this question is unclear to me. It's unclear where the doubt is. It's unclear why you're asking this question. Still, it makes us think why. So, why not challenge the status quo?  
  > **username_0**: @nbro I didn't originally think of weights in this way. I'm a visual learner and trying to think of real-world things I could compare weights to to help me visualize them and this idea came up.  
  > **username_3**: Simply, the answer is "no".  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_9 on 2023-09-04. Score: 8
  
  No, the weights do not need to add up to one. There isn't really a reason to do that. Weights as "contributions" may not be the best way to think about things here -- you're trying to learn a *function* that transforms the input values into your desired output values. Say you're learning a linear model and your data is 2D and drawn from a line with slope 2, then you'd want your learned weight to be 2.
  
  Additionally, say you have a feature that is *negatively* correlated with the output. That is, that a higher value in the input results in a lower outputs value. If all your weights are non-negative, you wouldn't be able to encode this at all!
  
  Finally, as the output would simply be a weighted average of the inputs, the *scale* of the outputs would be constrained by the scale of the inputs. Consider:
  $$f(\mathbf{x}) = \Sigma^d\_{i=1} w\_i x\_i$$
  with
  $$\Sigma^d\_{i=1} w\_i = 1$$
  
  $f(\mathbf{x})$ will always be less than or equal to $\max(\{w\_1, ..., w\_d\})$. For a simple linear model, you can definitely think of examples where the scale of the output is much larger than that of the input features (for example, number of bathrooms in a house versus its price). Although you can mitigate some of this by rescaling your data (as I will talk about shortly), there is really no reason to artificially constrain your model like this.
  
  A deep neural network can be thought of as stacks of these linear models, with the output of the previous layer being the input of the next layer. Although, these intuitions aren't at all perfectly transferable to deep neural networks, we would still be constraining the expressiveness of the model in a similar way: an increase in an input dimension can't result in a decrease in the output value and the scale of the output is constrained by the scale of the input.
  
  ## Normalization in Neural Networks
  
  You *do*, however, often need to perform normalization, but this normalization is done on the data and the outputs of hidden layers instead of the weights. If you add a weight penalty to your linear model for example, you want to make sure that each dimension is in around the same range, to not overly penalize features that tend to have larger weights. In neural networks, you also often normalize weight values for hidden layers, the most common variant is *Batch Normalization*.
  
  However, the normalization often used in these cases is a bit different from what you mentioned. Also known as *standardization*, this kind of normalization subtracts each sample by the mean (over a set of samples) and divides by the standard deviation (over that same set of samples).
  
  The exact reason why this is effective in neural networks is [an active area of research](https://arxiv.org/abs/1805.11604). A common explanation is [batch normalization reduces the effects of internal covariant shift](https://en.wikipedia.org/wiki/Batch_normalization), where changes in weights during training results in changes in the output distribution of a layer, which can harm performance as the next layer "expects" the original distribution. Centering and scaling the input to have zero mean and unit standard deviation reduces these changes. This, however, is disputed by [this paper](https://arxiv.org/pdf/1805.11604.pdf), which argues that batch normalization helps by making the loss landscape smoother.
  
  
  ---
  
  ### Answer by username_4 on 2023-09-04. Score: 6
  
  Nothing to do with it! Let me illustrate this concept using one of the simplest models in machine learning: Linear Regression. In linear regression, the goal is to find the coefficients of a hyperplane that best fits the given data, represented by the equation $Ax=b$. Here, we have free variables that represent the slopes along each axis and an independent term for the height (that in neural nets is called the Bias). Now, to address your question: Do the coefficients need to add up to 1? The answer is no.
  
  You might be tempted to interpret the concept of weights with the concept of probabilities, **but they are not the same!** They are merely coefficients in a high-dimensional, nested function.
  
  [![enter image description here](https://i.sstatic.net/x4JIY.png)](https://i.sstatic.net/x4JIY.png)
  
  Here's a 3D example of linear regression. The blue points represent the data, and the red plane is the best-fitting hyperplane according to the linear regression model.
  
  The equation of the fitted plane is approximately:
  
  $y=2.79⋅x\_1+2.01⋅x\_2+4.11$
  
  As you can see, the coefficients are $2.79$ for $x\_1$ $2.01$ for $x\_2$ , and the intercept is $4.11$. If we sum these together, we get $2.79 + 2.01 + 4.11 = 8.91$, which is clearly not equal to $1$.
  
  > **Comments:**
  > **username_5**: Couldn't you simply reparameterise the coefficients to [0.2817, 0.2026, 0.4157] and have them add up to 1 again?  
  > **username_6**: I think the main point here is the term "weights" has nothing to do with the term "weights" which represents a percentage.  
  > **username_4**: @ToivoSäwén If we parameterize the coefficients to the values you mentioned in order for them to add up to one, we will not end up with the same hyperplane. In the example I gave, we had the following linear equation:  y = 2.79x₁ + 2.01x₂ + 4.11  If we divide the entire equation by 8.91, we get:  y' = 0.31x₁ + 0.225x₂ + 0.416  This changes the y value to y'. Therefore, we would have to change our data "y" value in order for it to work. I believe you are confusing normalizing a vector with the property that all its components add up to one.  
  
  ---
  
  ### Answer by username_7 on 2023-09-04. Score: 3
  
  No, weights in that case do not constitute a formal probability distribution and **do not need to be add up to 1**. In fact, if they would — it could cause problems with underflow. Imagine a (nonsensical) neural network — *n* inputs, followed by 10 layers, each containing 10k parameters. This MLP would possess 100,000 parameters, excluding biases and the output layer. Given the novel architectures, it is not even a large number of parameters. Now, we normalize all the weights (layer or network-wise) so they form a valid probability distribution. It is easy to see, that the forward and backward passes could be problematic as the value of each parameter would be close to 0. Also, each gradient could be negligible small - a phenomenon that is observable now in NN due to various reasons [Vanishing Gradient Problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem).
  
  **How do we interpret the activity of the Neural Networks**? It is a reasonable question — there is a field of research that is called an [Explainable Artificial Intelligence](https://en.wikipedia.org/wiki/Explainable_artificial_intelligence) and deals with the interpretation of many deep learning architectures. Given that deep learning basically provides for creating a sequence of nested functions that can approximate arbitrary difficult non-linear functions it is a non-trivial task. If the weights would sum up to 1 — it could greatly help the progress. Unfortunately, this is not the case ;)
  
  
  ---
  
  ### Answer by username_8 on 2023-09-04. Score: 3
  
  > do all the weights need to add up to 1.0 (i.e. 100%)?
  
  No.
  
  > If not, why?
  
  Why should they?
  
  In the case of school grades, the justification might be that each component (midterm exam, homework etc.) is graded separately on a same scale and the final grade, which **must** lie on the same scale, is computed as the weighted average of these components.
  
  There is no such requirement regarding weights in neural networks.
  
  What you *might* want to enforce is that the outputs in the last layer sum to one, to obtain probabilities of class memberships (this, obviously, applies only to classification problems and not to regression). Softmax is typically used for that purpose.
  
  
  ---
  
  