Answers.Body: <p>(Partial) differential equations describe physical laws in many scientific domains. The purpose of <a href="https://en.wikipedia.org/wiki/Physics-informed_neural_networks" rel="nofollow noreferrer">Physics-informed Neural Networks (PINN)</a> is to incorporate <a href="https://en.wikipedia.org/wiki/Prior_probability" rel="nofollow noreferrer">prior information</a> from physical laws into neural network learning so they can better model physical systems. Modeling physical systems is solving differential equations.</p>
  <h1>Example: Airplane</h1>
  <p>For example, assume you must determine how air flows around an airplane wing at different speeds, given some data gathered in wind tunnel experiments. The result you want is a system that takes a time-varying speed as input and generates a function describing the movement of the air over time. Since differential equations govern air flow, the function we seek is a solution to those differential equations. Traditional methods to solve these equations are costly and require assumptions or measurements of boundary conditions. Training a PINN with our wind tunnel data will create a solution that not only fits our data but also obeys the laws we know apply to airflow. A conventional neural network trained with that same data will be less accurate because it can generate solutions that don't take into account the knowledge we have about airflow. It relies only on the data we give it, so it has to search a larger space, and it can generate physically impossible solutions.</p>
  <h1>Example: Toy problem</h1>
  <p>Imagine you have time-position values as follows:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Time (t)</th>
  <th>Position (y)</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>0</td>
  <td>0.1</td>
  </tr>
  <tr>
  <td>1</td>
  <td>0.9</td>
  </tr>
  <tr>
  <td>2</td>
  <td>4.1</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>This is <span class="math-container">$y=t^2 \pm 0.1$</span> - the position of an object under constant acceleration starting at the origin with a measurement error of <span class="math-container">$0.1$</span>.</p>
  <p>I trained a conventional neural network on this data with absolute error:</p>
  <p><span class="math-container">$\hat{y}=w_{3,1} f(w_{2,1,1} f(w_{1,1} t+b_{1,1}) + w_{2,1,2} f(w_{1,2} t+b_{1,2}) + b_{2,1}) + b_{3,1}$</span></p>
  <p>where <span class="math-container">$f$</span> is the ReLU activation function <span class="math-container">$f(x)=max(0, x)$</span></p>
  <p>This produces the following trained network:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Parameter</th>
  <th>Value</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td><span class="math-container">$weight_{1,1}$</span></td>
  <td>2.031941197</td>
  </tr>
  <tr>
  <td><span class="math-container">$weight_{1,2}$</span></td>
  <td>3.192294059</td>
  </tr>
  <tr>
  <td><span class="math-container">$bias_{1,1}$</span></td>
  <td>-0.617635647</td>
  </tr>
  <tr>
  <td><span class="math-container">$bias_{1,2}$</span></td>
  <td>-1.30428564</td>
  </tr>
  <tr>
  <td><span class="math-container">$weight_{2,1,1}$</span></td>
  <td>1.548372668</td>
  </tr>
  <tr>
  <td><span class="math-container">$weight_{2,1,2}$</span></td>
  <td>1.693544207</td>
  </tr>
  <tr>
  <td><span class="math-container">$bias_{2,1}$</span></td>
  <td>-3.251014905</td>
  </tr>
  <tr>
  <td><span class="math-container">$weight_{3,1}$</span></td>
  <td>0.374160198</td>
  </tr>
  <tr>
  <td><span class="math-container">$bias_{3,1}$</span></td>
  <td>0.100685864</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>If we instead use the differential equation for constant acceleration starting from rest, <span class="math-container">$\frac{dy}{dt} = a t$</span>, we can solve that by hand (thus a toy problem, you'd use PINN when you can't do this) to get <span class="math-container">$y=\frac{a t^2}{2} + b$</span>. You can absorb the constant into the weight to get <span class="math-container">$y=w t^2 + b$</span>. If I train this on the same data, I get:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>PDE Param</th>
  <th>Value</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>weight</td>
  <td>1.000157219</td>
  </tr>
  <tr>
  <td>boundary</td>
  <td>0.099369488</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>This yields the following predictions (including some held-out test data in bold italics)</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Time</th>
  <th>Measured Position</th>
  <th>NN Pred</th>
  <th>DE Pred</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>0</td>
  <td>0.1</td>
  <td>0.100685864</td>
  <td>0.099369488</td>
  </tr>
  <tr>
  <td>1</td>
  <td>0.9</td>
  <td>0.899997887</td>
  <td>1.099526707</td>
  </tr>
  <tr>
  <td>2</td>
  <td>4.1</td>
  <td>4.100000482</td>
  <td>4.099998364</td>
  </tr>
  <tr>
  <td><em><strong>3</strong></em></td>
  <td><em><strong>8.9</strong></em></td>
  <td><em><strong>7.300003076</strong></em></td>
  <td><em><strong>9.100784458</strong></em></td>
  </tr>
  <tr>
  <td><em><strong>4</strong></em></td>
  <td><em><strong>16.1</strong></em></td>
  <td><em><strong>10.50000567</strong></em></td>
  <td><em><strong>16.10188499</strong></em></td>
  </tr>
  <tr>
  <td><em><strong>5</strong></em></td>
  <td><em><strong>24.9</strong></em></td>
  <td><em><strong>13.70000827</strong></em></td>
  <td><em><strong>25.10329996</strong></em></td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Or in a plot:</p>
  <p><a href="https://i.sstatic.net/CFVnR.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CFVnR.png" alt="The neural network outputs are close for the first three points used in training but then diverge for the later points used in testing. The differential equation is so close as to be indistinguishable" /></a></p>
  <p>The conventional neural network is actually closer than the differential equation on the training data (the diagram is misleading on the second training point because of errors in the plotting software). But, when predicting data not in the initial training set, it is much worse because it does not know the appropriate underlying physical law.</p>
  <p>One can also see that the DE-based solution, though it is much closer, does not have the correct parameter values because it fits some measurement error. (It has a training error of <span class="math-container">$0.200158855$</span> and the correct solution has a training error of <span class="math-container">$0.3$</span>.) A full Bayesian treatment (beyond what PINNs do) would account for the uncertainty from measurement and limited data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-08T04:44:17.513Z
Answers.Text: Thanks for the detail answer! Reading your answer, I thought that PINN can be used in any equation if we can get the derivative, like y = m * exp(-b). Did I understood right?
Answers.UserDisplayName: COTHE
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-08T21:19:20.510Z
Answers.Text: The goal would be any equation where the integral is defined. However, that is a simplification. PINN is one method of solving DE using data. Current implementations have many cases where they become inefficient (for example, you might get the wrong answer, or it might not converge because you don't have infinite computation or you used the wrong approximation). See [the "Limitations" section of the Wikipedia article](https://en.wikipedia.org/wiki/Physics-informed_neural_networks#Limitations) for some known problem cases.
Answers.UserDisplayName: Eponymous
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-05T15:38:25.457Z
Answers.LastActivityDate: 2024-01-05T15:38:25.457Z
Answers.UserDisplayName: Eponymous
Body: <p>Im recently studying Physics informed neural networks.</p>
  <p>I found that it is designed for solving Differential equations.</p>
  <p>Is there any reason why it is used for differential equations only?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-05T13:07:55.017Z
Comments.Text: because ODE are very unstable learnt trivially by neural network, and the "physics informed" ones seem to behave better... but not, they can be applied in other areas
Comments.UserDisplayName: Alberto
Tags: <neural-networks><deep-learning>
Title: PINN - Why Differential equations only?
UserDisplayName: COTHE
ThreadText: # PINN - Why Differential equations only?
  
  (Asked by: username_0 on 2024-01-05)
  
  Im recently studying Physics informed neural networks.
  
  I found that it is designed for solving Differential equations.
  
  Is there any reason why it is used for differential equations only?
  
  **Tags:** neural-networks, deep-learning
  > **Comments:**
  > **username_1**: because ODE are very unstable learnt trivially by neural network, and the "physics informed" ones seem to behave better... but not, they can be applied in other areas  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-01-05. Score: 0 | Accepted answer
  
  (Partial) differential equations describe physical laws in many scientific domains. The purpose of [Physics-informed Neural Networks (PINN)](https://en.wikipedia.org/wiki/Physics-informed_neural_networks) is to incorporate [prior information](https://en.wikipedia.org/wiki/Prior_probability) from physical laws into neural network learning so they can better model physical systems. Modeling physical systems is solving differential equations.
  
  # Example: Airplane
  
  For example, assume you must determine how air flows around an airplane wing at different speeds, given some data gathered in wind tunnel experiments. The result you want is a system that takes a time-varying speed as input and generates a function describing the movement of the air over time. Since differential equations govern air flow, the function we seek is a solution to those differential equations. Traditional methods to solve these equations are costly and require assumptions or measurements of boundary conditions. Training a PINN with our wind tunnel data will create a solution that not only fits our data but also obeys the laws we know apply to airflow. A conventional neural network trained with that same data will be less accurate because it can generate solutions that don't take into account the knowledge we have about airflow. It relies only on the data we give it, so it has to search a larger space, and it can generate physically impossible solutions.
  
  # Example: Toy problem
  
  Imagine you have time-position values as follows:
  
  | Time (t) | Position (y) |
  | --- | --- |
  | 0 | 0.1 |
  | 1 | 0.9 |
  | 2 | 4.1 |
  
  This is $y=t^2 \pm 0.1$ - the position of an object under constant acceleration starting at the origin with a measurement error of $0.1$.
  
  I trained a conventional neural network on this data with absolute error:
  
  $\hat{y}=w\_{3,1} f(w\_{2,1,1} f(w\_{1,1} t+b\_{1,1}) + w\_{2,1,2} f(w\_{1,2} t+b\_{1,2}) + b\_{2,1}) + b\_{3,1}$
  
  where $f$ is the ReLU activation function $f(x)=max(0, x)$
  
  This produces the following trained network:
  
  | Parameter | Value |
  | --- | --- |
  | $weight\_{1,1}$ | 2.031941197 |
  | $weight\_{1,2}$ | 3.192294059 |
  | $bias\_{1,1}$ | -0.617635647 |
  | $bias\_{1,2}$ | -1.30428564 |
  | $weight\_{2,1,1}$ | 1.548372668 |
  | $weight\_{2,1,2}$ | 1.693544207 |
  | $bias\_{2,1}$ | -3.251014905 |
  | $weight\_{3,1}$ | 0.374160198 |
  | $bias\_{3,1}$ | 0.100685864 |
  
  If we instead use the differential equation for constant acceleration starting from rest, $\frac{dy}{dt} = a t$, we can solve that by hand (thus a toy problem, you'd use PINN when you can't do this) to get $y=\frac{a t^2}{2} + b$. You can absorb the constant into the weight to get $y=w t^2 + b$. If I train this on the same data, I get:
  
  | PDE Param | Value |
  | --- | --- |
  | weight | 1.000157219 |
  | boundary | 0.099369488 |
  
  This yields the following predictions (including some held-out test data in bold italics)
  
  | Time | Measured Position | NN Pred | DE Pred |
  | --- | --- | --- | --- |
  | 0 | 0.1 | 0.100685864 | 0.099369488 |
  | 1 | 0.9 | 0.899997887 | 1.099526707 |
  | 2 | 4.1 | 4.100000482 | 4.099998364 |
  | ***3*** | ***8.9*** | ***7.300003076*** | ***9.100784458*** |
  | ***4*** | ***16.1*** | ***10.50000567*** | ***16.10188499*** |
  | ***5*** | ***24.9*** | ***13.70000827*** | ***25.10329996*** |
  
  Or in a plot:
  
  [![The neural network outputs are close for the first three points used in training but then diverge for the later points used in testing. The differential equation is so close as to be indistinguishable](https://i.sstatic.net/CFVnR.png)](https://i.sstatic.net/CFVnR.png)
  
  The conventional neural network is actually closer than the differential equation on the training data (the diagram is misleading on the second training point because of errors in the plotting software). But, when predicting data not in the initial training set, it is much worse because it does not know the appropriate underlying physical law.
  
  One can also see that the DE-based solution, though it is much closer, does not have the correct parameter values because it fits some measurement error. (It has a training error of $0.200158855$ and the correct solution has a training error of $0.3$.) A full Bayesian treatment (beyond what PINNs do) would account for the uncertainty from measurement and limited data.
  
  > **Comments:**
  > **username_0**: Thanks for the detail answer! Reading your answer, I thought that PINN can be used in any equation if we can get the derivative, like y = m * exp(-b). Did I understood right?  
  > **username_2**: The goal would be any equation where the integral is defined. However, that is a simplification. PINN is one method of solving DE using data. Current implementations have many cases where they become inefficient (for example, you might get the wrong answer, or it might not converge because you don't have infinite computation or you used the wrong approximation). See [the "Limitations" section of the Wikipedia article](https://en.wikipedia.org/wiki/Physics-informed_neural_networks#Limitations) for some known problem cases.  
  
  ---
  
  