summaries: In this paper the authors inspect the popular Physics Informed Neural Network (PINN) set of models and characterize their behavior (specifically their failure modes) in the context of two popular PDE systems, namely diffusion and convection. In each case, the authors demonstrate that PINNs learn effective representations in simple regimes while failing to learn representations of the PDE systems in more complex regimes. In addition, authors also demonstrate that it is not the inability of the neural network to fit the functions (due to some potential inability to represent non-trivial functions), but rather the way the PINNs incorporate the PDEs into the learning pipeline that is the reason for the failure. Finally, the authors also propose casting the PINN problem as a sequential prediction task (i.e., predicting all collocation points per time period) as opposed to tasking the neural network with simultaneously learning the full spatial and temporal domain.   This work studies two specific (yet essential) PDE learning cases in the PINN framework. Notably, the work proposes two main conclusions to explain the difficult training in PINN networks:  - the loss landscape varies greatly with the PDE coefficient values (and works well with only low diffusion/conduction coefficient),  - the loss landscape (at the end of training) varies greatly with the physics informed penalty coefficient.   The main explanation to such difficulties lies in the condition number associated to PINN penalty. The work proposes two solutions: a warm-up, and a sequence to sequence approach. The paper studies the behavior of Physics Informed NNs (PINNs) on two common PDEs of physical relevance. It observes that PINNs fail to learn a good solution under standard training regime when the convection coefficient or viscosity coefficient is high, even when the closed-form solution does not depend on the constants.  The authors then seek to explain this behavior from an optimization standpoint. They plot the loss landscape along the two dominant hessian eigenvectors. They hypothesize that the optimization problem is ill conditioned due to the regularization term in the optimization objective that enforces the pde constraint.  They also run experiments to argue that sequence to sequence learning might work better for these ill conditioned problems. The authors provide a detailed empirical analysis of the numerical difficulties encountered when training so-called "physics informed neural networks" (PINNs), a recent class of models which incorporate physical (often differential) equations into the loss function of a neural network. The authors perform a systematic analysis on two relatively simple but common use-cases in scientific literature, convection and diffusion equations, and demonstrate the PINNs fail to adequately learn the underlying dynamics of the system even in simple cases. They show that these failures are due to the effect of the convection/diffusion coefficients and forcing terms on the conditioning of the dynamical system, i.e. larger coefficients or forcings tend to make the problem increasingly ill-conditioned, thereby impacting characteristics of the loss function. The authors then propose two possible workarounds for these issues; improved initialization via a warm start or preconditioning and re-framing the problem as a sequence-to-sequence learning task.
abstractText: Recent work in scientific machine learning has developed so-called physicsinformed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce a number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN’s setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN’s loss term starts from a simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.
authors.name: Aditi S. Krishnapriyan
authors.name: Amir Gholami
authors.name: Shandian Zhe
authors.name: Robert M. Kirby
authors.name: Michael W. Mahoney
references.authors: Y. Bengio J. Louradour R. Collobert J. Weston
references.title: Curriculum learning
references.venue: Proceedings of the 26th annual international conference on machine learning, pages 41–48
references.authors: S.L. Brunton B.R. Noack P. Koumoutsakos
references.title: Machine learning for fluid mechanics
references.venue: Annual Review of Fluid Mechanics, 52:477–508
references.authors: Y. Chen L. Lu G.E. Karniadakis L. Dal Negro
references.title: Physics-informed neural networks for inverse problems in nano-optics and metamaterials
references.venue: Optics express, 28(8):11618–11633
references.authors: M. Cranmer S. Greydanus S. Hoyer P. Battaglia D. Spergel S. Ho
references.title: Lagrangian neural networks
references.venue: arXiv preprint arXiv:2003.04630
references.authors: M. Dissanayake N. Phan-Thien
references.title: Neural-network-based approximations for solving partial differential equations
references.venue: communications in Numerical Methods in Engineering, 10(3):195–201
references.authors: P.L. Donti D. Rolnick J.Z. Kolter
references.title: Dc3: A learning method for optimization with hard constraints
references.venue: arXiv preprint arXiv:2104.12225
references.authors: V. Dwivedi N. Parashar B. Srinivasan
references.title: Distributed learning machines for solving forward and inverse problems in partial differential equations
references.venue: Neurocomputing, 420:299–316
references.authors: K. Eriksson D. Estep P. Hansbo C. Johnson
references.title: Computational differential equations
references.venue: Cambridge University Press
references.authors: B. Fornberg
references.title: A practical guide to pseudospectral methods
references.venue: Cambridge university press
references.authors: O. Fuks H.A. Tchelepi
references.title: Limitations of physics informed machine learning for nonlinear twophase transport in porous media
references.venue: Journal of Machine Learning for Modeling and Computing, 1 (1)
references.authors: N. Geneva N. Zabaras
references.title: Modeling the dynamics of pde systems with physics-constrained deep auto-regressive networks
references.venue: Journal of Computational Physics, 403:109056
references.authors: S. Greydanus M. Dzamba J. Yosinski
references.title: Hamiltonian neural networks
references.venue: Advances in Neural Information Processing Systems, 32:15379–15389
references.authors: J. Han A. Jentzen E. Weinan
references.title: Solving high-dimensional partial differential equations using deep learning
references.venue: Proceedings of the National Academy of Sciences, 115(34):8505–8510
references.authors: O. Hennigh S. Narasimhan M.A. Nabian A. Subramaniam K. Tangsali Z. Fang M. Rietmann W. Byeon S. Choudhry
references.title: Nvidia simnetTM: An ai-accelerated multi-physics simulation framework
references.venue: International Conference on Computational Science, pages 447–461. Springer
references.authors: W. Ji W. Qiu Z. Shi S. Pan S. Deng
references.title: Stiff-pinn: Physics-informed neural network for stiff chemical kinetics
references.venue: arXiv preprint arXiv:2011.04520
references.authors: X. Jin S. Cai H. Li G.E. Karniadakis
references.title: Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations
references.venue: Journal of Computational Physics, 426:109951
references.authors: G.E. Karniadakis I.G. Kevrekidis L. Lu P. Perdikaris S. Wang L. Yang
references.title: Physicsinformed machine learning
references.venue: Nature Reviews Physics, 3(6):422–440
references.authors: I.E. Lagaris A. Likas D.I. Fotiadis
references.title: Artificial neural networks for solving ordinary and partial differential equations
references.venue: IEEE transactions on neural networks, 9(5):987–1000
references.authors: Z. Long Y. Lu X. Ma B. Dong
references.title: Pde-net: Learning pdes from data
references.venue: International Conference on Machine Learning, pages 3208–3216. PMLR
references.authors: L. Lu X. Meng Z. Mao G.E. Karniadakis
references.title: Deepxde: A deep learning library for solving differential equations
references.venue: SIAM Review, 63(1):208–228
references.authors: L. Lu R. Pestourie W. Yao Z. Wang F. Verdugo S.G. Johnson
references.title: Physics-informed neural networks with hard constraints for inverse design
references.venue: arXiv preprint arXiv:2102.04626
references.authors: P. Márquez-Neila M. Salzmann P. Fua
references.title: Imposing hard constraints on deep networks: Promises and limitations
references.venue: arXiv preprint arXiv:1706.02025
references.authors: P. Moin
references.title: Fundamentals of engineering numerical analysis
references.venue: Cambridge University Press
references.authors: Y. Nandwani A. Pathak P. Singla
references.title: et al
references.venue: A primal dual formulation for deep learning with constraints. Advances in Neural Information Processing Systems
references.authors: D.R. Parisi M.C. Mariani M.A. Laborde
references.title: Solving differential equations with unsupervised neural networks
references.venue: Chemical Engineering and Processing: Process Intensification, 42(8-9):715– 721
references.authors: C. Rackauckas Y. Ma J. Martensen C. Warner K. Zubov R. Supekar D. Skinner A. Ramadhan A. Edelman
references.title: Universal differential equations for scientific machine learning
references.venue: arXiv preprint arXiv:2001.04385
references.authors: M. Raissi P. Perdikaris G.E. Karniadakis
references.title: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
references.venue: Journal of Computational Physics, 378:686–707
references.authors: M. Raissi A. Yazdani G.E. Karniadakis
references.title: Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations
references.venue: Science, 367(6481):1026–1030
references.authors: F. Sahli Costabal Y. Yang P. Perdikaris D.E. Hurtado E. Kuhl
references.title: Physics-informed neural networks for cardiac activation mapping
references.venue: Frontiers in Physics, 8:42
references.authors: J. Sirignano K. Spiliopoulos
references.title: Dgm: A deep learning algorithm for solving partial differential equations
references.venue: Journal of computational physics, 375:1339–1364
references.authors: B.P. van Milligen V. Tribaldos J. Jiménez
references.title: Neural network differential equation and plasma equilibrium solver
references.venue: Physical review letters,
references.authors: L. von Rueden S. Mayer K. Beckh B. Georgiev S. Giesselbach R. Heese B. Kirsch J. Pfrommer A. Pick R. Ramamurthy
references.title: Informed machine learning–a taxonomy and survey of integrating knowledge into learning systems
references.venue: arXiv preprint arXiv:1903.12394,
references.authors: B. Wang W. Zhang W. Cai
references.title: Multi-scale deep neural network (mscalednn) methods for oscillatory stokes flows in complex domains
references.venue: arXiv preprint arXiv:2009.12729
references.authors: S. Wang Y. Teng P. Perdikaris
references.title: Understanding and mitigating gradient pathologies in physics-informed neural networks
references.venue: arXiv preprint arXiv:2001.04536
references.authors: S. Wang H. Wang P. Perdikaris
references.title: On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks
references.venue: arXiv preprint arXiv:2012.10047
references.authors: S. Wang X. Yu P. Perdikaris
references.title: When and why pinns fail to train: A neural tangent kernel perspective
references.venue: arXiv preprint arXiv:2007.14527
references.authors: E. Weinan J. Han A. Jentzen
references.title: Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations
references.venue: Communications in Mathematics and Statistics, 5(4):349–380
references.authors: J. Willard X. Jia S. Xu M. Steinbach V. Kumar
references.title: Integrating physics-based modeling with machine learning: A survey
references.venue: arXiv preprint arXiv:2003.04919
references.authors: K. Xu E. Darve
references.title: Physics constrained learning for data-driven inverse modeling from sparse observations
references.venue: arXiv preprint arXiv:2002.10521
references.authors: Z. Yao A. Gholami Q. Lei K. Keutzer M.W. Mahoney
references.title: Hessian-based analysis of large batch training and robustness to adversaries
references.venue: Advances in Neural Information Processing Systems
references.authors: Z. Yao A. Gholami K. Keutzer M.W. Mahoney
references.title: Pyhessian: Neural networks through the lens of the hessian
references.venue: 2020 IEEE International Conference on Big Data (Big Data), pages 581–590. IEEE
references.authors: Y. Zhu N. Zabaras P.-S. Koutsourelakis P. Perdikaris
references.title: Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data
references.venue: Journal of Computational Physics, 394:56–81
references.authors: O.C. Zienkiewicz R.L. Taylor P. Nithiarasu J. Zhu
references.title: The finite element method
references.venue: volume 3. McGraw-hill London
sections.heading: 1 Introduction
sections.text: Partial differential equations (PDEs) are commonly used to describe different phenomena in science and engineering. These PDEs are often derived by starting from governing first principles (e.g., conservation of mass or energy). It is typically not possible to find analytical solutions to these PDEs for many real-world settings. Thus, many different numerical methods (e.g., the finite element method [44], pseudo-spectral methods [9], etc.) have been introduced to approximate their solutions/behavior. However, these PDEs can be quite complex for several settings (e.g., turbulence simulations), and numerical integration techniques, which typically update and improve a candidate solution iteratively until convergence, are often quite computationally expensive. Motivated by this—as well as the increasing quantities of data available in many scientific and engineering applications—there has been recent interest in developing machine learning (ML) approaches to find the solution of the underlying PDEs (and/or work in tandem with numerical solutions). As a result, the area of Scientific Machine Learning (SciML)—which aims to couple traditional scientific mechanistic modeling
  ∗Equal contribution
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  (typically, differential equations) with data-driven ML methodologies (most recently, neural network training)—has emerged. In this vein, there have been a number of ML approaches to incorporate scientific knowledge into such problems while keeping the automatic, data-driven estimates of the solution [2, 17, 33, 39].
  A recent line of work involves Physics-Informed Neural Network (PINN) models, which aim to incorporate physical domain knowledge as soft constraints on an empirical loss function, that is then optimized using existing ML training methodologies. To some degree, PINNs are an example of “grafting together” domain-driven models and data-driven methodologies. However, there are important subtleties with this, and we identify several possible failure modes with a naive approach. We then illustrate possible directions for addressing these failure modes.
  Background and problem overview. Many of the problems with a PDE constraint fit the following abstraction: F(u(x, t)) = 0, x ∈ Ω ⊂ Rd, t ∈ [0, T ], (1) where F is a differential operator representing the PDE, u(x, t) is the state variable (i.e., parameter of interest), x/t denote space/time, T is the time horizon, and Ω is the spatial domain. Since F is a differential operator, in general one must specify appropriate boundary and/or initial conditions to ensure the existence/uniqueness of a solution to Eq. 1. In the context of PDEs, F can be taxonomized into a parabolic, hyperbolic, or elliptic differential operator [23]. Quintessential examples of F include: the convection equation (a hyperbolic PDE), where u(x, t) could model fluid movement, e.g., air or some liquid, over space and time; the diffusion equation (a parabolic PDE), where u(x, t) could model the temperature distribution over space and time; and the Laplace equation (an elliptic PDE), where u(x) could model a steady-state diffusion equation, in the limit as t→∞. One possible data-driven approach is to incorporate domain information by applying Eq. 1 as a “hard constraint” when training a NN on the data. This can be formulated as the following constrained optimization problem,
  min θ L(u) s.t. F(u) = 0, (2)
  where L(u) is the data-fit term (including initial/boundary conditions), and where F is a constraint on the residual of the PDE system under consideration (i.e., the “physics” knowledge in the equation itself). As mentioned before, for many practical use cases, it is not possible to derive closed form solutions for these problems, and it is often quite difficult to solve problems of the form of Eq. 2, with F(u) as a hard constraint. Another (related but different) data-driven approach is to impose the constraint as a “soft constraint” on the outputs of the NN model,
  min θ L(u) + λFF(u), (3)
  L(u) = Lu0 + Lub . (4) Here, Lu0 and Lub measure the misfit of the NN prediction and the initial/boundary conditions (which are pre-specified/given as input to the problem), and θ denotes the NN parameters (which takes (x, t), and possibly other quantities, as inputs and then outputs u(x, t)). Furthermore, λF is a regularization parameter that controls the emphasis on the PDE based residual (which we ideally want to be zero). The goal is then to use ML methodologies (stochastic optimization, etc.) to train this NN model to minimize the loss in Eq. 3. In particular, the NN is trained to minimize this modified loss function, where the modification is to penalize the violations of F(u) for some λF ≥ 0. However, even with a large training dataset, this approach does not guarantee that the NN will obey the conservation/governing equations in the constraint Eq. 1. In many SciML problems, these sorts of constraints on the system matter, as they correspond to physical mechanisms of the system. For example, if the conservation of energy equation is only approximately satisfied, then the system being simulated may behave qualitatively differently or even result in unrealistic solutions.
  We should also note that this approach of incorporating physics-based regularization, where the regularization constraint, LF , corresponds to a differential operator, is very different than incorporating much simpler norm-based regularization (such as L1 or L2 regularization), as is common in ML more generally. Here, the regularization operator, LF is non-trivially structured—it involves a differential operator that could actually be ill-conditioned, and it does not correspond to a nice convex set (as does
  a norm ball). Moreover, LF corresponds to actual physical quantities, and there is often an important distinction between satisfying the constraint exactly versus satisfying the constraint approximately (the soft constraint approach doing only the latter).
  Main contributions. The contributions of this paper are as follows:
  • We analyze PINN models on simple, yet physically relevant, problems of convection, reaction, and reaction-diffusion. We find that the vanilla/regular PINN approach only works for very easy parameter regimes (i.e., small PDE coefficients), but that it fails to learn relevant physics in even moderately more challenging physical regimes, even for problems that have simple closed-form analytical solutions. For many cases, the vanilla PINN approach achieves almost 100% error, as compared to the ground truth solution, even after extensive hyperparameter tuning. (See §3 for details.)
  • We analyze the loss landscape of trained PINN models and find that adding/increasing the PDE-based soft constraint regularization (LF in Eq. 3) makes it more complex and harder to optimize, especially for cases with non-trivial coefficients. We also study how the loss landscape changes as the regularization parameter (λF ) is changed. We find that reducing the regularization parameter can help alleviate the complexity of the loss landscape, but this in turn leads to poor solutions with high errors that do not satisfy the PDE/constraint. (See §4 for details.)
  • We demonstrate that the NN architecture has the capacity/expressivity to find a good solution, thereby showing that these problems are not due to the limited capacity of the NN architecture. Instead, we argue that the failure is due to optimization difficulties associated with the PINN’s soft PDE constraint. (See §5 for details.)
  • We propose two paths forward to address these failure modes through (i) curriculum regularization and (ii) posing the learning problem as a sequence-to-sequence learning task. First, in curriculum regularization, we start by imposing the PDE constraint (LF ) with small coefficients, which are progressively increased to the target problem’s settings as the model gets trained. This gives the NN an opportunity to first train with easier constraints, before it is exposed to the target constraint which could be hard to optimize from the beginning. Second, we show that changing the learning problem to a sequence-to-sequence learning problem can reduce the PINN error, again without any change to the NN architecture. In this setup, the NN is trained on a time segment, instead of the full space-time, which could be more difficult to learn. The task is then to predict the solution and reduce the loss only over smaller time segments. We extensively test both approaches and show that they can reduce the error by up to 1-2 orders of magnitude as compared to regular PINN training, and in many cases can better capture “sharp” features in the solution. (See §5 for details.)
  • We have open sourced our framework [26] which is built on top of PyTorch both to help with reproducibility and also to enable other researchers to extend the results.
sections.heading: 2 Related work
sections.text: There is a large body of related work, and here we briefly discuss the most related lines of work.
  Machine learning and PDEs. ML approaches for PDE problems have been increasing rapidly in recent years [13, 19]. A number of tools and methodologies now exist to solve scientific problems by combining ML and domain insights [14, 20, 27, 28, 38]. As mentioned earlier, a popular approach to combine ML and physical knowledge is to include aspects of the PDE term as part of the optimization process via regularization. A notable aspect of such an approach is that the NN can be trained only on data that comes from the governing equation(s) itself (though additional data can be included as well, if available), i.e., with a relatively small amount of data. This has garnered interest and shown successful results in a wide variety of science and engineering problems and applications [3, 11, 16, 29–31, 43].
  However, there have also been issues observed with this formulation. For example, it did not work well for stiff ordinary differential equations (ODEs) describing chemical kinetics [15], for certain heterogeneous media problems [7], or for certain fluid flow problems [10]. Furthermore, PINN models have been analyzed in the context of neural tangent kernels (i.e., towards the infinite width
  limit) to study their convergence [36, 37]. This work found some cases where the model failed (such as when the target function exhibits “high frequency features”) and showed some preliminary solutions via the lens of the neural tangent kernel. It has been argued that some of these problems may be due an imbalance in back-propagated gradients in the loss function during training, and a learning-rate annealing scheme has been proposed to mitigate this [35].
  Physical priors and constraints in NNs. Imposing physical priors and constraints on NN systems is common in SciML problems, as a way to try to enforce a property of interest. This idea has been introduced in different forms in the past (for instance [5, 18, 25, 28, 32]). Some approaches have focused on embedding specialized physical constraints into NNs, such as conservation of energy or momentum [4, 12] or multiscale features [34]. While methods focusing on constraining the output of the NN are more common, it is difficult to enforce such constraints exactly in ML settings. Previous work has tried to impose hard constraints in ML (both within the context of SciML and otherwise) [6, 21, 22, 24, 40], although this can be computationally expensive, and does not guarantee better results or convergence.
sections.heading: 3 Possible failure modes for physics-informed neural networks
sections.text: In this section, we highlight several examples where the PINN formulation defined in Eq. 3 does not predict the solution well. We first demonstrate this with two different types of simple, canonical PDE/ODE systems which have simple analytical solutions: convection ( §3.1), and reaction ( §A). We then also include a diffusion component by looking at the reaction-diffusion problem ( §3.2). Note that the convection problem has a linear PDE constraint, and reaction/reaction-diffusion problems both have non-linear PDE terms.2 We show that PINNs can only learn simple problems with very small parameter values (e.g., small convection or reaction coefficients). We demonstrate that these models fail to learn the relevant physical phenomena for non-trivial cases (e.g., relatively larger coefficients). As we will see, while adding the physical constraint as a soft regularization may be easier to deploy and optimize with existing unconstrained optimization methods, this approach does come with trade-offs, including that in many cases the optimization problem becomes much more difficult to solve.
  Experiment setup. We study both linear and non-linear PDEs/ODEs, and we vary the convection, reaction, and diffusion coefficients for each problem (hereafter, we refer to these as PDE coefficients). For each problem, we aim to minimize the loss function in Eq. 3. We use a 4-layer fully-connected NN with 50 neurons per layer, a hyperbolic tangent activation function, and randomly sample collocation points (x, t) on the domain. Furthermore, all the systems that we consider have periodic boundary conditions. We enforce this through an extra term in the loss function that takes the difference between the predicted NN solution at each boundary. We train this network using the L-BFGS optimizer and sweep over learning rates from 1e−4 to 2.0.3 After training the PINN, we measure the L2 relative and absolute errors between the PINN’s predicted solution and the analytical solution. The L2 relative error is 1
  N
  ∑N i=0
  || û− u ||2 || u ||2 ; and the absolute error is 1 N
  ∑N i=0 || û− u ||2, where
  N is the number of evaluation grid points, û is the predicted solution by the PINN, and u is the true solution. For all cases, we run models at least ten times with different preset random seeds, and we average the relative and absolute errors in u(x, t). For each loss function, û is the output of the NN and shorthand for û = NN(θ, x, t).
  2Note that for convection, this does not necessarily mean that the mapping from initial to final solution is also linear. This just means that the terms in the PDE are linear.
  3For reasons that are only partially understood, L-BFGS methods tend to perform better for existing PINN problems. While variants of stochastic gradient descent are much more popular in computer vision, natural language processing, and recommendation systems, we found that they underperform in comparison to L-BFGS.
sections.heading: 3.1 Learning convection
sections.text: Problem formulation. We first consider a one-dimensional convection problem, a hyperbolic PDE which is commonly used to model transport phenomena:
  ∂u ∂t + β ∂u ∂x = 0, x ∈ Ω, t ∈ [0, T ], (5)
  u(x, 0) = h(x), x ∈ Ω. Here, β is the convection coefficient and h(x) is the initial condition. For constant β and periodic boundary conditions, this problem has a simple analytical solution:
  uanalytical(x, t) = F −1(F (h(x))e−iβkt), (6)
  where F is the Fourier transform, i = √ −1, and k denotes frequency in the Fourier domain. The general loss function for this problem (corresponding to Eq. 3) is
  L(θ) = 1 Nu Nu∑ i=1 ( û− ui0 )2 + 1 Nf Nf∑ i=1 λi (∂û ∂t + β ∂û ∂x )2 + LB, (7)
  where û = NN(θ, x, t) is the output of the NN, and LB is the boundary loss. For periodic boundary conditions with Ω = [0, 2π), this loss is:
  LB = 1
  Nb Nb∑ i=1 ( û(θ, 0, t)− û(θ, 2π, t) )2 . (8)
  We use the following simple initial and periodic boundary conditions:
  u(x, 0) = sin(x),
  u(0, t) = u(2π, t). (9)
  Observations. We apply the PINN’s soft regularization to this problem, and we optimize the loss function in Eq. 7. After training, we measure the relative and absolute errors between the PINN’s predicted solution and the analytical solution, as reported in Fig. 1(a). As one can see, the PINN is only able to achieve good solutions for small values of convection coefficient, and it fails when β becomes larger, reaching a relative error of almost 100% for β > 10. We also provide visualization of the exact and PINN solution in Fig. 1(b-c). One can clearly see that the PINN is unable to learn the solution. As we will later show, the NN architecture does have enough capacity to find the solution, but the training/optimization problem is very difficult to solve with PINNs (and importantly, it may require extensive hyperparameter tuning which is often not feasible in practice).
sections.heading: 3.2 Learning reaction-diffusion
sections.text: Problem formulation. We next look at a reaction-diffusion system, where we add a diffusion operator to the reaction equation discussed above. Note that for pure diffusion, the solution dissipates
  to a steady-state of uniform/constant distribution, which may be trivial to learn. Therefore, we consider studying the reaction-diffusion system:
  ∂u ∂t − ν ∂
  2u
  ∂x2 − ρu(1− u) = 0, x ∈ Ω, t ∈ (0, T ], (10)
  u(x, 0) = h(x), x ∈ Ω. Here, ν (ν > 0) is the diffusion coefficient. The solution of such a system can be solved for via Strang splitting, i.e., splitting the equation into two separate models (a reaction component and a diffusion component):
  du dt = ρu(1− u) du
  dt = ν
  ∂2u ∂x2 .
  (11)
  For each timestep, we can solve the reaction equation through Eq. 15 (in §A). The diffusion equation has the following analytical solution:
  uanalytical(x, t) = F −1(F (u(x, t = tn))e−νk2t), (12)
  where u(x, t = tn) is the solution at the nth time step. We solve the reaction equation for each timestep, and then use the reaction solution as the initial condition to solve the diffusion component and get the final solution.
  The general loss function for this problem is,
  L(θ) = 1 Nu Nu∑ i=1 ( û− ui0 )2 +
  1
  Nf Nf∑ i=1 λi (∂û ∂t − ν ∂ 2û ∂x2 − ρû(1− û) )2 + LB,
  (13)
  where LB is the boundary loss. Similar to the previous example, periodic boundary conditions can be enforced by including LB from Eq. 8 as an extra term in the loss.
  Observations. Similar to the previous case, we can see that the PINN also fails to learn reactiondiffusion. We illustrate a case in Fig. 2 with ρ = 5, when ν = 5. The PINN achieves a high relative error of 93%. Here, we can clearly see that the PINN is unable to capture either the reaction or diffusion component. Additional figures for different ν values can be seen in Fig. D.1. In particular, for ν = 2 the PINN achieves a relative error of 50%. Here, we see that it is unable to capture the “sharper” transitions, though it can predict the center of the solution a little better.
sections.heading: 4 Diagnosing possible failure modes for physics-informed NNs
sections.text: Thus far, we have shown that PINNs can result in high errors even for simple physical regimes, in particular for PDEs/ODEs with non-trivial convection/reaction/diffusion coefficients. Here, we
  demonstrate that one of the underlying reasons for this arises due to the PDE-based soft constraint of LF , which makes the loss landscape difficult to optimize. We first (in §4.1) analyze the loss landscape to illustrate how increasing this soft regularization can lead to more complex loss landscapes, thus leading to optimization difficulties. We then (in §B) demonstrate how this is related to regularizing with differential operators, which can result in ill-conditioning.
sections.heading: 4.1 Soft PDE regularization and optimization difficulties
sections.text: Here, we analyze how the loss landscape changes for different regimes for the convection problem in §3.1 with/without the soft regularization in PINNs. We show that adding the soft regularization can actually make the problem harder to optimize, i.e., the regularization leads to less smooth loss landscapes. For all the experiments, we plot the loss landscape by perturbing the (trained) model across the first two dominant Hessian eigenvectors and computing the corresponding loss values. This tends to be more informative than perturbing the model parameters in random directions [41, 42].
  Figure 3 shows the loss landscape for the convection problem (discussed in §3.1), for different β values. Interestingly, the loss landscape at a relatively low β = 1 is rather smooth, but increasing β further results in a complex and non-symmetric loss landscape. It is also evident that the optimizer has gotten stuck in a local minima with a very high loss function for large β values.
  Finally, we study the impact of changing the weight/multiplier for the soft regularization term (i.e., the λ parameter in Eq. 3), which can be relevant in improving PINN performance [35]. While we find that tuning λ can help change the error, it cannot resolve the problem, as shown in Fig. E.1. Note that as the regularization parameter is increased, the loss landscape becomes increasingly more complex and harder to optimize (additionally, see the z-axis scale).
sections.heading: 5 Expressivity versus optimization difficulty
sections.text: In this section, we first show that the failure modes we observed are not necessarily due to the specific NN architecture that we used in our experiments. In particular, we show that the NN model does have the expressivity/capacity to learn the convection/reaction/diffusion coefficient cases where the vanilla PINN method fails. Additionally, in the process of demonstrating this, we also describe two methods that lead to significantly lower error rates. In particular, we show that changing the learning paradigm to curriculum regularization can make the optimization problem easier to solve (as discussed in §5.1). Second, we show that posing the problem as sequence-to-sequence learning may lead to better results than learning the entire state-space at once (as discussed in §5.2).
sections.heading: 5.1 Curriculum PINN Regularization
sections.text: One may contend that the failure modes shown in §3 may be because the NN does not have enough capacity. Here, we show that this is not the underlying reason. To do so, we devise a “curriculum
  regularization” method to warm start the NN training by finding a good initialization for the weights. Instead of training the PINN to learn the solution right away for cases with higher β/ρ, we start by training the PINN on lower β/ρ (easier for the PINN to learn) and then gradually move to training the PINN on higher β/ρ, respectively. We test these results for the examples in §3.1 and §A. This is somewhat analogous to curriculum learning in ML [1], but applied by progressively making the PDE/ODE harder to solve.
  Figure 4 shows the training procedure for an example convection case ( §3.1) with β = 30. As Fig. 4(c) shows, the curriculum regularization approach results in a much more accurate solution than regular PINN training. With curriculum regularization, the relative error is almost two orders of magnitude lower. Additionally, this is true across all the other regimes that we found regular PINNs to fail, as shown in Tab. 1. In Fig. E.2, we also show that curriculum regularization not only decreases error significantly, but also decreases the variance of the error. In Fig. E.3, we see that curriculum regularization results in a much smoother loss landscape as compared to regular PINN training.
  Curriculum regularization also works well for the reaction example in §A. In this case, we start by training with a low ρ value (reaction coefficient), and then increase gradually to higher ρ values. The results can be seen in Fig. E.4. We can see that the error is 0.1 - 0.6 orders of magnitude lower for ρ = 2− 4 (when the regular PINN error is not as high), and then greatly decreases error by 1-2 orders of magnitude for ρ = 5− 10. As we discussed before, PINN has difficulty in learning sharp features for high values of ρ. However, the curriculum regularization overcomes this, even for ρ = 10, as seen in Fig. E.4(c).
sections.heading: 5.2 Sequence-to-sequence learning vs learning the entire space-time solution
sections.text: The original PINN approach of [28] trains the NN model to predict the entire space-time at once (i.e., predict u for all locations and time points). In certain cases, this can be more difficult to learn. Here, we demonstrate that it may be better to pose the problem as a sequence-to-sequence (seq2seq) learning task, where the NN learns to predict the solution at the next time step, instead of all times.
  0.0 0.2 0.4 0.6 0.8 1.0 t
  0
  1
  2
  3
  4
  5
  6
  x
  Initial condition points Boundary points Collocation points
  (a) Regular PINN training
  t 2 t 3 t 4 t 5 t 6 t 7 t 8 t 9 t 10 t t 0
  1
  2
  3
  4
  5
  6
  x
  (b) Sequence-to-sequence learning (model trained every ∆t)
  0.0 0.2 0.4 0.6 0.8 1.0 t
  0
  1
  2
  3
  4
  5
  6
  x
  Initial condition points Boundary points Collocation points
  Figure 5: Schematic outlining seq2seq learning. In contrast to regular PINN training, the solution in seq2seq learning is predicted for only one ∆t step at a time. Then, the predicted solution at t = ∆t is used as the initial condition for the next segment. To allow fair comparison, we keep the total number of collocation points to be exactly the same in either approach. That is, we do not increase the number of collocation points for seq2seq learning in the right, and keep it to be the same as in the corresponding segment in the left figure.
  Entire state space ∆t = 0.05 ∆t = 0.1 ν = 2, ρ = 5 Relative error 5.07× 10−1 2.04× 10−2 1.18× 10−2
  Absolute error 2.70× 10−1 1.06× 10−2 6.41× 10−3
  ν = 3, ρ = 5 Relative error 7.98× 10−1 1.92× 10−2 1.56× 10−2 Absolute error 4.79× 10−1 1.01× 10−2 8.17× 10−3
  ν = 4, ρ = 5 Relative error 8.84× 10−1 2.37× 10−2 1.59× 10−2 Absolute error 5.74× 10−1 1.15× 10−2 8.01× 10−3
  ν = 5, ρ = 5 Relative error 9.35× 10−1 2.36× 10−2 2.39× 10−2 Absolute error 6.46× 10−1 1.09× 10−2 1.15× 10−2
  ν = 6, ρ = 5 Relative error 9.60× 10−1 2.81× 10−2 2.69× 10−2 Absolute error 6.84× 10−1 1.17× 10−2 1.28× 10−2
  Table 2: Predicting the entire state space versus discretizing the state space (i.e., seq2seq learning) for 1D reaction-diffusion ( §3.2). The seq2seq learning achieves lower error for both ∆t = 0.05 and ∆t = 0.1, in comparison to the PINN’s approach of predicting the entire state space at once.
  This way, we can use a marching-in-time scheme to predict different sequences/time points. Note that the only data available here is from the PDE itself, i.e., just the initial condition. We take the prediction at t = ∆t and use this as the initial condition to make a prediction at t = 2∆t, and so on. This is schematically outlined in Fig. 5.
  We test this scheme by using the exact same NN architecture as in previous sections, and we report the results in Tab. E.1 for the convection problem of §3.1, Tab. E.2 for the reaction problem of §A, and Tab. 2 for the reaction-diffusion problem of §3.2. We compare the relative/absolute error when the learning is posed as a seq2seq problem (i.e., predicting the state space with a “time marching scheme” of one timestep prediction at a time) to the PINN approach of predicting the whole state space at once.4
  We explore the following cases where the PINN does poorly, varying β, ρ, and ν coefficients:
  4To have a fair comparison between the two methods (time marching versus predicting entire state space at once), for the time marching method, we use the same number of collocation (interior) points for both. For example, for T = [0, 1], if we use 1000 collocation points to predict the entire state space, then for ∆t = 0.1 we use 100 collocation points per section.
  1) For 1D convection ( §3.1), higher β values from 30-40.
  2) For 1D reaction ( §A), ρ coefficients from 5-10.
  3) For 1D reaction-diffusion ( §3.2), a fixed ρ = 5 and ν coefficients from 2-6.
  For these cases, we find that posing the problem as seq2seq learning results in significantly lower error. The difference is particularly striking for the reaction and reaction-diffusion cases, where the seq2seq PINN model decreases error by almost two orders of magnitude. An example case is shown Fig. 6, where the seq2seq approach is able to recover the solution, while regular PINNs does very poorly. Note that this behavior also has analogues with numerical methods used in scientific computing, where space-time problems are typically harder to solve, as compared to time marching methods [8]. Intuitively, since the problem is ill-conditioned, restricting the dimensions is expected to help. Furthermore, the underlying function/mapping of the input to the solution should be much simpler to approximate over a smaller time span, as compared to the full time horizon.
  These initial results are promising, and further developments may lead to still better ways of using PINNs and learning PDEs. In particular, using more sophisticated methods to predict timesteps across the state space may provide improved performance, as may including more sophisticated seq2seq approaches and tuning the regularization parameter (i.e., amount of constraint added).
sections.heading: 6 Conclusions
sections.text: PINNs—and SciML more generally—hold great promise for expanding the scope of ML methodology to important problems in science and engineering. For these problems, however, integrating ML methods with PDE-based domain-driven constraints as a soft regularization term can lead to subtle and critical issues. In particular, we show that this approach can have fundamental limitations which results in failure modes for learning relevant physics commonly used in different fields of science. To show this, we picked two fundamental PDE problems of diffusion and convection and showed that the PINN only works for very simple cases, failing to learn the relevant physical phenomena for even moderately more challenging regimes. We then analyzed the problem to characterize the underlying reasons why these failures occur. In particular, we studied the PINN loss landscape behavior and found it becomes it becomes increasingly complex for large values of diffusion or convection coefficients, and with/without non-homogeneous forcing. We also discussed that the problem is not necessarily due to the limited capacity of the NN, but that it is partly an optimization problem resulting in the PDE-based soft constraint used in PINNs. Furthermore, we showed that the PINN approach of solving for the entire space-time at once may not be efficient, and instead posing the problem as a sequence-to-sequence learning task can provide lower error rates. Addressing these and related issues will be critical if we hope to go beyond existing cut-and-paste approaches, toward engineering a more intimate connection between scientific methodologies and ML methodologies. This will be needed to deliver on the promise of PINNs and SciML more generally.
sections.heading: 7 Acknowledgements.
sections.text: We are thankful to Shashank Subramanian for his feedback and contributions. We also acknowledge helpful discussions with Prof. George Biros, Geoffrey Negiar, and Daniel Rothchild. ASK was supported by Laboratory Directed Research and Development (LDRD) funding under Contract Number DE-AC02-05CH11231 at LBNL and the Alvarez Fellowship in the Computational Research Division at LBNL. AG was supported through funding from Samsung SAIT. MWM would also like to acknowledge the UC Berkeley CLTC, ARO, NSF, and ONR. The UC Berkeley team also acknowledges gracious support from Intel corporation, Intel VLAB, Samsung, Amazon AWS, Google Cloud, Google TPU Research Cloud, and Google Brain (in particular Prof. David Patterson, Dr. Ed Chi, and Jing Li). Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.
title: Characterizing possible failure modes in physics-informed neural networks