<p>I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For example, let <span class="math-container">$(W^k_{ij})_{ij}$</span> be the weights of the (dense) <span class="math-container">$k$</span>-th layer.
Are there architectures where it is imposed something like
<span class="math-container">$$
\sum_{i, j} (W^k_{ij})^2 = 1
$$</span>
(namely the roll-out vector of weights is constrained to stay on a sphere) or <span class="math-container">$W^k_{ij}$</span> are equivalence classes <span class="math-container">$mod K$</span> for some number <span class="math-container">$K&gt;0$</span>?</p>
<p>Then, of course, one should probably think about proper activation functions for these cases, but it's probably not a big obstacle.</p>
<p>Putting constraints of these kinds will prevent the weights to grow indefinitely and maybe could prevent over-fitting?</p>

CC BY-SA 4.0
2021-11-25T22:00:53.353Z
Hello. Welcome to Artificial Intelligence Stack Exchange. I put what I think is your main specific question in the title. Make sure that's correct. Having said that, you tagged your post with [tag:weight-normalization], so I suppose you're aware of those techniques. Right?
nbro
CC BY-SA 4.0
2021-11-26T07:07:42.343Z
Hi @nbro and thanks for the correction! Not really! I used that tag because it was recommended when I start typing "weight"
Onil90
<neural-networks><deep-learning><deep-neural-networks><weights><weight-normalization>
Are there neural networks with (hard) constraints on the weights?
Onil90
# Are there neural networks with (hard) constraints on the weights?

(Asked by: username_0 on 2021-11-25)

I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For example, let $(W^k\_{ij})\_{ij}$ be the weights of the (dense) $k$-th layer.
Are there architectures where it is imposed something like
$$
\sum\_{i, j} (W^k\_{ij})^2 = 1
$$
(namely the roll-out vector of weights is constrained to stay on a sphere) or $W^k\_{ij}$ are equivalence classes $mod K$ for some number $K>0$?

Then, of course, one should probably think about proper activation functions for these cases, but it's probably not a big obstacle.

Putting constraints of these kinds will prevent the weights to grow indefinitely and maybe could prevent over-fitting?

**Tags:** neural-networks, deep-learning, deep-neural-networks, weights, weight-normalization

> **Comments:**
> **username_1**: Hello. Welcome to Artificial Intelligence Stack Exchange. I put what I think is your main specific question in the title. Make sure that's correct. Having said that, you tagged your post with [tag:weight-normalization], so I suppose you're aware of those techniques. Right?  
> **username_0**: Hi @username_1 and thanks for the correction! Not really! I used that tag because it was recommended when I start typing "weight"  

---

## 0 answers

