Challenges in Training PINNs: A Loss Landscape Perspective
<|reference_start|>Challenges in Training PINNs: A Loss Landscape Perspective: This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill-conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, showing the superiority of Adam+L-BFGS, and introduce a novel second-order optimizer, NysNewton-CG (NNCG), which significantly improves PINN performance. Theoretically, our work elucidates the connection between ill-conditioned differential operators and ill-conditioning in the PINN loss and shows the benefits of combining first- and second-order optimization methods. Our work presents valuable insights and more powerful optimization strategies for training PINNs, which could improve the utility of PINNs for solving difficult partial differential equations.<|reference_end|>
@article{rathore2024challenges,
    title={Challenges in Training PINNs: A Loss Landscape Perspective},
    author={Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, Madeleine Udell},
    journal={arXiv preprint arXiv:2402.01868},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.01868},
    primaryClass={cs.LG math.OC stat.ML}
}