Answers.Body: <p>Some ideas out the top of my head:</p>
  
  <ol>
  <li><p>In the case of <span class="math-container">$dy/dx_2&gt;0$</span> you could compute the gradient using the chain rule and limit the weights so that the constrain holds</p></li>
  <li><p>In the case of <span class="math-container">$y + x_5 + x_7 &lt; K$</span> you could use a clipping function on the output layer?</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-18T08:29:30.043Z
Answers.LastActivityDate: 2019-11-18T08:29:30.043Z
Answers.UserDisplayName: Javier-Acuna
Body: <p>I'm trying to train a neural network to do a multiple non-linear regression <span class="math-container">$y=f(x_i), i=1,2…N$</span>. So far it works good (low MSE), but some predictions <span class="math-container">$y$</span> are “non-physical”, for instance for our application it is known  from first principles that when <span class="math-container">$x_2$</span> increases, then <span class="math-container">$y$</span> also has to increase (<span class="math-container">$dy/dx_2&gt;0$</span>), but in some instances the neural network’s output doesn’t comply with this constraint. Another example is that <span class="math-container">$y + x_5 + x_7$</span> should be less than a constant <span class="math-container">$K$</span></p>
  
  <p>I thought about adding a penalty term to the loss function to enforce these constraints, but I am wondering if there is a "harder" way to impose such a constraint (that is, to ensure that these constraints will always hold, no only that non-physical predictions will be penalized)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-14T11:05:23.280Z
Comments.Text: About the second one, is it x5+x7 <K or f(x5)+f(x7)<K? The former is not a constraint on the output but and error in the dataset. You should be able to weed them out without machine learning.
Comments.UserDisplayName: serali
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-14T11:15:00.287Z
Comments.Text: Hi serali, I had a typo in that equation, I meant the constaint y + x_5 + x_7 < K
Comments.UserDisplayName: Ken Grimes
Tags: <neural-networks><machine-learning><regression>
Title: Imposing physical constraints (previous knowledge) in a neural network for regression
UserDisplayName: Ken Grimes
ThreadText: # Imposing physical constraints (previous knowledge) in a neural network for regression
  
  (Asked by: username_0 on 2019-11-14)
  
  I'm trying to train a neural network to do a multiple non-linear regression $y=f(x\_i), i=1,2…N$. So far it works good (low MSE), but some predictions $y$ are “non-physical”, for instance for our application it is known from first principles that when $x\_2$ increases, then $y$ also has to increase ($dy/dx\_2>0$), but in some instances the neural network’s output doesn’t comply with this constraint. Another example is that $y + x\_5 + x\_7$ should be less than a constant $K$
  
  I thought about adding a penalty term to the loss function to enforce these constraints, but I am wondering if there is a "harder" way to impose such a constraint (that is, to ensure that these constraints will always hold, no only that non-physical predictions will be penalized)
  
  **Tags:** neural-networks, machine-learning, regression
  > **Comments:**
  > **username_1**: About the second one, is it x5+x7 <K or f(x5)+f(x7)<K? The former is not a constraint on the output but and error in the dataset. You should be able to weed them out without machine learning.  
  > **username_0**: Hi serali, I had a typo in that equation, I meant the constaint y + x_5 + x_7 < K  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-11-18. Score: 2
  
  Some ideas out the top of my head:
  
  1. In the case of $dy/dx\_2>0$ you could compute the gradient using the chain rule and limit the weights so that the constrain holds
  2. In the case of $y + x\_5 + x\_7 < K$ you could use a clipping function on the output layer?
  
  
  ---
  
  