In-Context Learning of Physical Properties: Few-Shot Adaptation to Out-of-Distribution Molecular Graphs
<|reference_start|>In-Context Learning of Physical Properties: Few-Shot Adaptation to Out-of-Distribution Molecular Graphs: Large language models manifest the ability of few-shot adaptation to a sequence of provided examples. This behavior, known as in-context learning, allows for performing nontrivial machine learning tasks during inference only. In this work, we address the question: can we leverage in-context learning to predict out-of-distribution materials properties? However, this would not be possible for structure property prediction tasks unless an effective method is found to pass atomic-level geometric features to the transformer model. To address this problem, we employ a compound model in which GPT-2 acts on the output of geometry-aware graph neural networks to adapt in-context information. To demonstrate our model's capabilities, we partition the QM9 dataset into sequences of molecules that share a common substructure and use them for in-context learning. This approach significantly improves the performance of the model on out-of-distribution examples, surpassing the one of general graph neural network models.<|reference_end|>
@article{kaszuba2024in-context,
    title={In-Context Learning of Physical Properties: Few-Shot Adaptation to
  Out-of-Distribution Molecular Graphs},
    author={Grzegorz Kaszuba, Amirhossein D. Naghdi, Dario Massa, Stefanos
  Papanikolaou, Andrzej Jaszkiewicz, Piotr Sankowski},
    journal={arXiv preprint arXiv:2406.01808},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01808},
    primaryClass={cs.LG cond-mat.mtrl-sci}
}