Answers.Body: <p>Extending @mirror2image's comment, if you have a certain metric that allows you to measure how close the intended layer is to a low pass filter (something that compares its output with what a low pass filter would have produced, for example), the simplest way to achieve what you want would be to add a term in your loss function that calculates the value of this metric. This way, each time you do a training step, the network now is not only made to output the correct predictions but is also forced to do so while also keeping that specific layer's behavior as close to a low-pass filter as possible.
  This is the most common way of tweaking the behavior of neural networks and is often encountered in many research papers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-12T07:25:22.470Z
Answers.LastActivityDate: 2019-06-12T07:25:22.470Z
Answers.UserDisplayName: Suyog Jadhav
Answers.Body: <p>the idea about training that it is allowed for weights (not layer as you wrote) "learn" values what they "want" from general network setting. But actualy i v thinked about that too, and to have weights represent as you want, you can first train a shorter network, while having those weights (1) as very last, so you have maximum control over them, after trained a lot append next layers to those and learning rate for weights (1) make much smaller as for new weights </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-12T09:42:03.663Z
Answers.LastActivityDate: 2019-06-12T09:42:03.663Z
Answers.UserDisplayName: user8426627
Body: <p>In general, how does one make a neural network learn the training data while also forcing it to represent some known structure (e.g., representing a family of functions)?</p>
  <p>The neural network might find the optimal weights, but those weights might no longer make the layer represent the function I originally intended.</p>
  <p>For example, suppose I want to create a convolutional layer in the middle of my neural network that is a low-pass filter. In the context of the entire network, however, the layer might cease to be a low-pass filter at the end of training because the backpropagation algorithm found a better optimum.</p>
  <p>How do I allow the weights to be as optimal as possible, while still maintaining the low-pass characteristics I originally wanted?</p>
  <p>General tips or pointing to specific literature would be much appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-11T05:34:15.987Z
Comments.Text: Two ways to do it - freeze above mentioned layer, don't update it. Another - add regularizer to loss wich force that layer to retain it's properties. Latter could be very non-trivial
Comments.UserDisplayName: mirror2image
Tags: <neural-networks><reference-request><constrained-optimization>
Title: How does one make a neural network learn the training data while also forcing it to represent some known structure?
UserDisplayName: Alan
ThreadText: # How does one make a neural network learn the training data while also forcing it to represent some known structure?
  
  (Asked by: username_0 on 2019-06-10)
  
  In general, how does one make a neural network learn the training data while also forcing it to represent some known structure (e.g., representing a family of functions)?
  
  The neural network might find the optimal weights, but those weights might no longer make the layer represent the function I originally intended.
  
  For example, suppose I want to create a convolutional layer in the middle of my neural network that is a low-pass filter. In the context of the entire network, however, the layer might cease to be a low-pass filter at the end of training because the backpropagation algorithm found a better optimum.
  
  How do I allow the weights to be as optimal as possible, while still maintaining the low-pass characteristics I originally wanted?
  
  General tips or pointing to specific literature would be much appreciated.
  
  **Tags:** neural-networks, reference-request, constrained-optimization
  > **Comments:**
  > **username_1**: Two ways to do it - freeze above mentioned layer, don't update it. Another - add regularizer to loss wich force that layer to retain it's properties. Latter could be very non-trivial  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-06-12. Score: 2
  
  Extending @mirror2image's comment, if you have a certain metric that allows you to measure how close the intended layer is to a low pass filter (something that compares its output with what a low pass filter would have produced, for example), the simplest way to achieve what you want would be to add a term in your loss function that calculates the value of this metric. This way, each time you do a training step, the network now is not only made to output the correct predictions but is also forced to do so while also keeping that specific layer's behavior as close to a low-pass filter as possible.
  This is the most common way of tweaking the behavior of neural networks and is often encountered in many research papers.
  
  
  ---
  
  ### Answer by username_3 on 2019-06-12. Score: 0
  
  the idea about training that it is allowed for weights (not layer as you wrote) "learn" values what they "want" from general network setting. But actualy i v thinked about that too, and to have weights represent as you want, you can first train a shorter network, while having those weights (1) as very last, so you have maximum control over them, after trained a lot append next layers to those and learning rate for weights (1) make much smaller as for new weights
  
  
  ---
  
  