Answers.Body: <p>I, unfortunately, cannot provide you with a scientifically based answer, so I'll try to logic my way to an answer.</p>
  <p>I know that there are things called 'nonnegativity-constrained autoencoders'. I do not know the details of their initialization but you could look into those for possible guidelines. There are possibly also other networks which have papers published talking about a 'nonnegativity-constrained' variant which I am unaware of.</p>
  <p>Furthermore, I would have some concerns/thoughts about nonnegative neural nets:</p>
  <ol>
  <li>I would assume that having only positive network weights could lead to an exploding gradient. If anything, I would scale the input in a similar range as your weight initialization and have the upper bound of your weight initialization not exceed 1.</li>
  <li>Logically, but also building further on point 1, I would not use (variants of) ReLu activations. With non-negative constrained weights and non-negative inputs, ReLu becomes a simple linear activation. I would advise against any activation function without an upper limit, as it might again result in exploding weights/gradients.</li>
  <li>I would start testing the model with sigmoid activations in all layers (except possibly the last, of course).</li>
  <li>Xavier/glorot initialization is usually best for layers that also use negative weights and a tanh activation function. I do not know how a nonnegative weight constraint changes its effectiveness. However, if you centre your input around (i.e.) 0.5, taking a uniform distribution around 0.5 and vary its range depending on the layer size might work.</li>
  </ol>
  <p>Again these are just my insights, not necessarily based on scientific research.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T15:29:06.433Z
Answers.LastActivityDate: 2023-01-15T15:29:06.433Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I wonder how to initialize the weights of a layer with non-negative weight constraints and sigmoid activation afterwards. I did not find any guidelines. I thought about taking inspiration of glorotUniform initialization and using <span class="math-container">$\text{Uniform}[0,b]$</span> with <span class="math-container">$b=2 \sqrt{6 / (n_{in} + n_{out})}$</span> where <span class="math-container">$n_{in}$</span> and <span class="math-container">$n_{out}$</span> are the number of input and output units respectively. Another idea I had was choosing <span class="math-container">$a=\frac{n-12}{2n}$</span>, <span class="math-container">$b=1-\frac{n-12}{2n}$</span> where <span class="math-container">$n$</span> is the number of input units and sampling out of <span class="math-container">$\text{Uniform}[a,b]$</span> such that the variance of each output variable is equal to <span class="math-container">$1$</span>.</p>
  <p>My question: Are there any guidelines how to initialize non-negative weights? If no, do you think my suggestions run in problems?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-13T14:19:05.430Z
Comments.Text: Can you provide a bit more detail on why your weights have a non-negative constraint? Is that constraint enforced during optimization? Or is it only during initialization? What is your reason for wanting this?
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-13T14:28:11.237Z
Comments.Text: The input and output lives in a cone, i.e. multiplication with positive values stays in the cone as well as addition of two cone elements. In some layers I use specific cone-operations which require that the input is a cone element. Non-negative weight constraints can be easily implemented using e.g. constraints in tensorflow. Problem is that I don't know about initialization guidelines.
Comments.UserDisplayName: Philipp123
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-14T19:58:31.370Z
Comments.Text: I'm also curious: Do you have a reason to believe that it will benefit training to keep the weights $\gt 0$? To me it would make more sense to only constrain the network output like this. Regarding the question, I don't know about any guidelines, but I think your idea to augment the glorot uniform like this makes sense. Except that I'd choose $b = 0.5 \sqrt{6 / (n_{in} + n_{out})}$ to keep the layer output at a similar magnitude. Multiplying by two would yield outputs that are four times larger compared to the standard glorot-uniform-initialized layer, which might harm training.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-15T14:06:22.200Z
Comments.Text: @Chillston Thanks for the input, yes, positivity was more or less theoretically motivated because I'm working in the cone of positive definite symmetrical matrices and I thought about the values staying in it all the time but I also might try with negative values
Comments.UserDisplayName: Philipp123
Tags: <neural-networks><weights-initialization>
Title: Neural network: Initial weights for layer with non-negative constraint
UserDisplayName: Philipp123
ThreadText: # Neural network: Initial weights for layer with non-negative constraint
  
  (Asked by: username_0 on 2022-07-01)
  
  I wonder how to initialize the weights of a layer with non-negative weight constraints and sigmoid activation afterwards. I did not find any guidelines. I thought about taking inspiration of glorotUniform initialization and using $\text{Uniform}[0,b]$ with $b=2 \sqrt{6 / (n\_{in} + n\_{out})}$ where $n\_{in}$ and $n\_{out}$ are the number of input and output units respectively. Another idea I had was choosing $a=\frac{n-12}{2n}$, $b=1-\frac{n-12}{2n}$ where $n$ is the number of input units and sampling out of $\text{Uniform}[a,b]$ such that the variance of each output variable is equal to $1$.
  
  My question: Are there any guidelines how to initialize non-negative weights? If no, do you think my suggestions run in problems?
  
  **Tags:** neural-networks, weights-initialization
  > **Comments:**
  > **username_1**: Can you provide a bit more detail on why your weights have a non-negative constraint? Is that constraint enforced during optimization? Or is it only during initialization? What is your reason for wanting this?  
  > **username_0**: The input and output lives in a cone, i.e. multiplication with positive values stays in the cone as well as addition of two cone elements. In some layers I use specific cone-operations which require that the input is a cone element. Non-negative weight constraints can be easily implemented using e.g. constraints in tensorflow. Problem is that I don't know about initialization guidelines.  
  > **username_2**: I'm also curious: Do you have a reason to believe that it will benefit training to keep the weights $\gt 0$? To me it would make more sense to only constrain the network output like this. Regarding the question, I don't know about any guidelines, but I think your idea to augment the glorot uniform like this makes sense. Except that I'd choose $b = 0.5 \sqrt{6 / (n_{in} + n_{out})}$ to keep the layer output at a similar magnitude. Multiplying by two would yield outputs that are four times larger compared to the standard glorot-uniform-initialized layer, which might harm training.  
  > **username_0**: @Chillston Thanks for the input, yes, positivity was more or less theoretically motivated because I'm working in the cone of positive definite symmetrical matrices and I thought about the values staying in it all the time but I also might try with negative values  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-15. Score: 1
  
  I, unfortunately, cannot provide you with a scientifically based answer, so I'll try to logic my way to an answer.
  
  I know that there are things called 'nonnegativity-constrained autoencoders'. I do not know the details of their initialization but you could look into those for possible guidelines. There are possibly also other networks which have papers published talking about a 'nonnegativity-constrained' variant which I am unaware of.
  
  Furthermore, I would have some concerns/thoughts about nonnegative neural nets:
  
  1. I would assume that having only positive network weights could lead to an exploding gradient. If anything, I would scale the input in a similar range as your weight initialization and have the upper bound of your weight initialization not exceed 1.
  2. Logically, but also building further on point 1, I would not use (variants of) ReLu activations. With non-negative constrained weights and non-negative inputs, ReLu becomes a simple linear activation. I would advise against any activation function without an upper limit, as it might again result in exploding weights/gradients.
  3. I would start testing the model with sigmoid activations in all layers (except possibly the last, of course).
  4. Xavier/glorot initialization is usually best for layers that also use negative weights and a tanh activation function. I do not know how a nonnegative weight constraint changes its effectiveness. However, if you centre your input around (i.e.) 0.5, taking a uniform distribution around 0.5 and vary its range depending on the layer size might work.
  
  Again these are just my insights, not necessarily based on scientific research.
  
  
  ---
  
  