<p>I, unfortunately, cannot provide you with a scientifically based answer, so I'll try to logic my way to an answer.</p>
<p>I know that there are things called 'nonnegativity-constrained autoencoders'. I do not know the details of their initialization but you could look into those for possible guidelines. There are possibly also other networks which have papers published talking about a 'nonnegativity-constrained' variant which I am unaware of.</p>
<p>Furthermore, I would have some concerns/thoughts about nonnegative neural nets:</p>
<ol>
<li>I would assume that having only positive network weights could lead to an exploding gradient. If anything, I would scale the input in a similar range as your weight initialization and have the upper bound of your weight initialization not exceed 1.</li>
<li>Logically, but also building further on point 1, I would not use (variants of) ReLu activations. With non-negative constrained weights and non-negative inputs, ReLu becomes a simple linear activation. I would advise against any activation function without an upper limit, as it might again result in exploding weights/gradients.</li>
<li>I would start testing the model with sigmoid activations in all layers (except possibly the last, of course).</li>
<li>Xavier/glorot initialization is usually best for layers that also use negative weights and a tanh activation function. I do not know how a nonnegative weight constraint changes its effectiveness. However, if you centre your input around (i.e.) 0.5, taking a uniform distribution around 0.5 and vary its range depending on the layer size might work.</li>
</ol>
<p>Again these are just my insights, not necessarily based on scientific research.</p>

CC BY-SA 4.0
2023-01-15T15:29:06.433Z
2023-01-15T15:29:06.433Z
Robin van Hoorn
<p>I wonder how to initialize the weights of a layer with non-negative weight constraints and sigmoid activation afterwards. I did not find any guidelines. I thought about taking inspiration of glorotUniform initialization and using <span class="math-container">$\text{Uniform}[0,b]$</span> with <span class="math-container">$b=2 \sqrt{6 / (n_{in} + n_{out})}$</span> where <span class="math-container">$n_{in}$</span> and <span class="math-container">$n_{out}$</span> are the number of input and output units respectively. Another idea I had was choosing <span class="math-container">$a=\frac{n-12}{2n}$</span>, <span class="math-container">$b=1-\frac{n-12}{2n}$</span> where <span class="math-container">$n$</span> is the number of input units and sampling out of <span class="math-container">$\text{Uniform}[a,b]$</span> such that the variance of each output variable is equal to <span class="math-container">$1$</span>.</p>
<p>My question: Are there any guidelines how to initialize non-negative weights? If no, do you think my suggestions run in problems?</p>

CC BY-SA 4.0
2023-01-13T14:19:05.430Z
Can you provide a bit more detail on why your weights have a non-negative constraint? Is that constraint enforced during optimization? Or is it only during initialization? What is your reason for wanting this?
Robin van Hoorn
CC BY-SA 4.0
2023-01-13T14:28:11.237Z
The input and output lives in a cone, i.e. multiplication with positive values stays in the cone as well as addition of two cone elements. In some layers I use specific cone-operations which require that the input is a cone element. Non-negative weight constraints can be easily implemented using e.g. constraints in tensorflow. Problem is that I don't know about initialization guidelines.
Philipp123
CC BY-SA 4.0
2023-01-14T19:58:31.370Z
I'm also curious: Do you have a reason to believe that it will benefit training to keep the weights $\gt 0$? To me it would make more sense to only constrain the network output like this. Regarding the question, I don't know about any guidelines, but I think your idea to augment the glorot uniform like this makes sense. Except that I'd choose $b = 0.5 \sqrt{6 / (n_{in} + n_{out})}$ to keep the layer output at a similar magnitude. Multiplying by two would yield outputs that are four times larger compared to the standard glorot-uniform-initialized layer, which might harm training.
Chillston
CC BY-SA 4.0
2023-01-15T14:06:22.200Z
@Chillston Thanks for the input, yes, positivity was more or less theoretically motivated because I'm working in the cone of positive definite symmetrical matrices and I thought about the values staying in it all the time but I also might try with negative values
Philipp123
<neural-networks><weights-initialization>
Neural network: Initial weights for layer with non-negative constraint
Philipp123
# Neural network: Initial weights for layer with non-negative constraint

(Asked by: username_0 on 2022-07-01)

I wonder how to initialize the weights of a layer with non-negative weight constraints and sigmoid activation afterwards. I did not find any guidelines. I thought about taking inspiration of glorotUniform initialization and using $\text{Uniform}[0,b]$ with $b=2 \sqrt{6 / (n\_{in} + n\_{out})}$ where $n\_{in}$ and $n\_{out}$ are the number of input and output units respectively. Another idea I had was choosing $a=\frac{n-12}{2n}$, $b=1-\frac{n-12}{2n}$ where $n$ is the number of input units and sampling out of $\text{Uniform}[a,b]$ such that the variance of each output variable is equal to $1$.

My question: Are there any guidelines how to initialize non-negative weights? If no, do you think my suggestions run in problems?

**Tags:** neural-networks, weights-initialization

> **Comments:**
> **username_1**: Can you provide a bit more detail on why your weights have a non-negative constraint? Is that constraint enforced during optimization? Or is it only during initialization? What is your reason for wanting this?  
> **username_0**: The input and output lives in a cone, i.e. multiplication with positive values stays in the cone as well as addition of two cone elements. In some layers I use specific cone-operations which require that the input is a cone element. Non-negative weight constraints can be easily implemented using e.g. constraints in tensorflow. Problem is that I don't know about initialization guidelines.  
> **username_2**: I'm also curious: Do you have a reason to believe that it will benefit training to keep the weights $\gt 0$? To me it would make more sense to only constrain the network output like this. Regarding the question, I don't know about any guidelines, but I think your idea to augment the glorot uniform like this makes sense. Except that I'd choose $b = 0.5 \sqrt{6 / (n_{in} + n_{out})}$ to keep the layer output at a similar magnitude. Multiplying by two would yield outputs that are four times larger compared to the standard glorot-uniform-initialized layer, which might harm training.  
> **username_0**: @username_2 Thanks for the input, yes, positivity was more or less theoretically motivated because I'm working in the cone of positive definite symmetrical matrices and I thought about the values staying in it all the time but I also might try with negative values  

---

## 1 answers

### Answer by username_1 on 2023-01-15. Score: 1

I, unfortunately, cannot provide you with a scientifically based answer, so I'll try to logic my way to an answer.

I know that there are things called 'nonnegativity-constrained autoencoders'. I do not know the details of their initialization but you could look into those for possible guidelines. There are possibly also other networks which have papers published talking about a 'nonnegativity-constrained' variant which I am unaware of.

Furthermore, I would have some concerns/thoughts about nonnegative neural nets:

1. I would assume that having only positive network weights could lead to an exploding gradient. If anything, I would scale the input in a similar range as your weight initialization and have the upper bound of your weight initialization not exceed 1.
2. Logically, but also building further on point 1, I would not use (variants of) ReLu activations. With non-negative constrained weights and non-negative inputs, ReLu becomes a simple linear activation. I would advise against any activation function without an upper limit, as it might again result in exploding weights/gradients.
3. I would start testing the model with sigmoid activations in all layers (except possibly the last, of course).
4. Xavier/glorot initialization is usually best for layers that also use negative weights and a tanh activation function. I do not know how a nonnegative weight constraint changes its effectiveness. However, if you centre your input around (i.e.) 0.5, taking a uniform distribution around 0.5 and vary its range depending on the layer size might work.

Again these are just my insights, not necessarily based on scientific research.


---

