Body: <p>I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For example, let <span class="math-container">$(W^k_{ij})_{ij}$</span> be the weights of the (dense) <span class="math-container">$k$</span>-th layer.
  Are there architectures where it is imposed something like
  <span class="math-container">$$
  \sum_{i, j} (W^k_{ij})^2 = 1
  $$</span>
  (namely the roll-out vector of weights is constrained to stay on a sphere) or <span class="math-container">$W^k_{ij}$</span> are equivalence classes <span class="math-container">$mod K$</span> for some number <span class="math-container">$K&gt;0$</span>?</p>
  <p>Then, of course, one should probably think about proper activation functions for these cases, but it's probably not a big obstacle.</p>
  <p>Putting constraints of these kinds will prevent the weights to grow indefinitely and maybe could prevent over-fitting?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-25T22:00:53.353Z
Comments.Text: Hello. Welcome to Artificial Intelligence Stack Exchange. I put what I think is your main specific question in the title. Make sure that's correct. Having said that, you tagged your post with [tag:weight-normalization], so I suppose you're aware of those techniques. Right?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-26T07:07:42.343Z
Comments.Text: Hi @nbro and thanks for the correction! Not really! I used that tag because it was recommended when I start typing "weight"
Comments.UserDisplayName: Onil90
Tags: <neural-networks><deep-learning><deep-neural-networks><weights><weight-normalization>
Title: Are there neural networks with (hard) constraints on the weights?
UserDisplayName: Onil90
ThreadText: # Are there neural networks with (hard) constraints on the weights?
  
  (Asked by: username_0 on 2021-11-25)
  
  I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For example, let $(W^k\_{ij})\_{ij}$ be the weights of the (dense) $k$-th layer.
  Are there architectures where it is imposed something like
  $$
  \sum\_{i, j} (W^k\_{ij})^2 = 1
  $$
  (namely the roll-out vector of weights is constrained to stay on a sphere) or $W^k\_{ij}$ are equivalence classes $mod K$ for some number $K>0$?
  
  Then, of course, one should probably think about proper activation functions for these cases, but it's probably not a big obstacle.
  
  Putting constraints of these kinds will prevent the weights to grow indefinitely and maybe could prevent over-fitting?
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks, weights, weight-normalization
  
  > **Comments:**
  > **username_1**: Hello. Welcome to Artificial Intelligence Stack Exchange. I put what I think is your main specific question in the title. Make sure that's correct. Having said that, you tagged your post with [tag:weight-normalization], so I suppose you're aware of those techniques. Right?  
  > **username_0**: Hi @username_1 and thanks for the correction! Not really! I used that tag because it was recommended when I start typing "weight"  
  
  ---
  
  ## 0 answers
  
  