MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data
<|reference_start|>MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data: Large language models (LLMs), like ChatGPT, have shown that even trained with noisy prior data, they can generalize effectively to new tasks through in-context learning (ICL) and pre-training techniques. Motivated by this, we explore whether a similar approach can be applied to scientific foundation models (SFMs). Our methodology is structured as follows: (i) we collect low-cost physics-informed neural network (PINN)-based approximated prior data in the form of solutions to partial differential equations (PDEs) constructed through an arbitrary linear combination of mathematical dictionaries; (ii) we utilize Transformer architectures with self and cross-attention mechanisms to predict PDE solutions without knowledge of the governing equations in a zero-shot setting; (iii) we provide experimental evidence on the one-dimensional convection-diffusion-reaction equation, which demonstrate that pre-training remains robust even with approximated prior data, with only marginal impacts on test accuracy. Notably, this finding opens the path to pre-training SFMs with realistic, low-cost data instead of (or in conjunction with) numerical high-cost data. These results support the conjecture that SFMs can improve in a manner similar to LLMs, where fully cleaning the vast set of sentences crawled from the Internet is nearly impossible.<|reference_end|>
@article{kang2024mad-scientist:,
    title={MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction
  Equations Using Massive PINN-Based Prior Data},
    author={Mingu Kang, Dongseok Lee, Woojin Cho, Jaehyeon Park, Kookjin Lee,
  Anthony Gruber, Youngjoon Hong, Noseong Park},
    journal={arXiv preprint arXiv:2410.06442},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.06442},
    primaryClass={cs.LG cs.AI}
}