Answers.Body: <p>I have a somewhat similar problem, there's this paper that I was supposed to extend (I can explain how): <a href="https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-017-1984-2" rel="nofollow noreferrer">https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-017-1984-2</a>
  What they do is that they have a connection network between input and second layer (giving some biological context to training) I don't know how you can maintain the zero edges during back propagation though.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-26T18:30:27.613Z
Answers.LastActivityDate: 2019-06-26T18:30:27.613Z
Answers.UserDisplayName: embi
Body: <p>Gradient training changes indiscriminately all the weights and nodes of the neural network. But one can imagine the situations when the training should be shaped, e.g.:</p>
  
  <ul>
  <li>One can put constraints on some of the weights. E.g. human brain contains regions whose inner connections are more dense that external connections with different regions. One can try to mimic this region-shaped structure in neural networks as well and hence one can require that inter-regional weights (in opposit to intra-regional weights) are close to zero (except, possibly, for some channels among regions);</li>
  <li>One can put constraints on some of the weights in such manner that some layer of neurons have specific structure. E.g. consider the popular encoder-decoder architecture of neural machine translation e.g. <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noreferrer">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a> We can see that that the whole output of the encoder is expressed as a single layer of neurons which is forwarded to the input of the decoder. So - one can require that the set of all the possible outputs of the encoder (e.g. the possible values of this single layer of the neurons) forms some kind of structure, e.g. some grammar of some <em>interlingua</em>. This example is for illustration only, I have in mind more complex neural network which has one layer of neurons which indeed should output the encoded words of some interlingua. So, one is required to guid all the weights of the encoder in such manner that this single layer has only allowable values.</li>
  </ul>
  
  <p>So - my question is - <strong>are there methods that guide the gradient descent training with additional information about the weights or about the nodes</strong> (i.e. about the whole subsets of weights that have some impact on specific layer of nodes)? E.g. <strong>about methods that impress the region structure on the neural network or that constrains the values of some nodes to be in specific range only?</strong></p>
  
  <p>Of course, it <strong>is quite easy to include such constraints in evolutionary neural networks</strong> - one can simply reject the neural networks with weights that violates the constraints. But <strong>is it possible to do this in gradient-like training?</strong></p>
  
Tags: <neural-networks><training><gradient-descent><evolutionary-algorithms>
Title: How to shape the weights or nodes during gradient training of neural network? Training with constraints?
UserDisplayName: TomR
ThreadText: # How to shape the weights or nodes during gradient training of neural network? Training with constraints?
  
  (Asked by: username_0 on 2019-01-30)
  
  Gradient training changes indiscriminately all the weights and nodes of the neural network. But one can imagine the situations when the training should be shaped, e.g.:
  
  * One can put constraints on some of the weights. E.g. human brain contains regions whose inner connections are more dense that external connections with different regions. One can try to mimic this region-shaped structure in neural networks as well and hence one can require that inter-regional weights (in opposit to intra-regional weights) are close to zero (except, possibly, for some channels among regions);
  * One can put constraints on some of the weights in such manner that some layer of neurons have specific structure. E.g. consider the popular encoder-decoder architecture of neural machine translation e.g. <https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html> We can see that that the whole output of the encoder is expressed as a single layer of neurons which is forwarded to the input of the decoder. So - one can require that the set of all the possible outputs of the encoder (e.g. the possible values of this single layer of the neurons) forms some kind of structure, e.g. some grammar of some *interlingua*. This example is for illustration only, I have in mind more complex neural network which has one layer of neurons which indeed should output the encoded words of some interlingua. So, one is required to guid all the weights of the encoder in such manner that this single layer has only allowable values.
  
  So - my question is - **are there methods that guide the gradient descent training with additional information about the weights or about the nodes** (i.e. about the whole subsets of weights that have some impact on specific layer of nodes)? E.g. **about methods that impress the region structure on the neural network or that constrains the values of some nodes to be in specific range only?**
  
  Of course, it **is quite easy to include such constraints in evolutionary neural networks** - one can simply reject the neural networks with weights that violates the constraints. But **is it possible to do this in gradient-like training?**
  
  **Tags:** neural-networks, training, gradient-descent, evolutionary-algorithms
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-26. Score: 1
  
  I have a somewhat similar problem, there's this paper that I was supposed to extend (I can explain how): <https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-017-1984-2>
  What they do is that they have a connection network between input and second layer (giving some biological context to training) I don't know how you can maintain the zero edges during back propagation though.
  
  
  ---
  
  