title: Towards Data-Efficient Pretraining for Atomic Property Prediction
summary: This paper challenges the recent paradigm in atomic property prediction that
  links progress to growing dataset sizes and computational resources. We show
  that pretraining on a carefully selected, task-relevant dataset can match or
  even surpass large-scale pretraining, while using as little as 1/24th of the
  computational cost. We introduce the Chemical Similarity Index (CSI), a novel
  metric inspired by computer vision's Fr\'echet Inception Distance, for
  molecular graphs which quantifies the alignment between upstream pretraining
  datasets and downstream tasks. By selecting the most relevant dataset with
  minimal CSI distance, we show that models pretrained on a smaller, focused
  dataset consistently outperform those pretrained on massive, mixed datasets
  such as JMP, even when those larger datasets include the relevant dataset.
  Counterintuitively, we also find that indiscriminately adding more data can
  degrade model performance when the additional data poorly aligns with the task
  at hand. Our findings highlight that quality often outperforms quantity in
  pretraining for atomic property prediction.
paper_authors._id: 67b44f44620ae0bad17d6699
paper_authors.name: Yasir Ghunaim
paper_authors._id: 67b44f44620ae0bad17d669a
paper_authors.name: Hasan Abed Al Kader Hammoud
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:30:43.057Z
paper_authors._id: 642b51385bf2355d02a23d15
paper_authors.avatarUrl: /avatars/87985347643b2647555f2453fa4d94fb.svg
paper_authors.fullname: Hasan Abed Al Kader Hammoud
paper_authors.type: user
paper_authors.user: hammh0a
paper_authors._id: 67b44f44620ae0bad17d669b
paper_authors.name: Bernard Ghanem
paper_title: Towards Data-Efficient Pretraining for Atomic Property Prediction
paper_summary: This paper challenges the recent paradigm in atomic property prediction that
  links progress to growing dataset sizes and computational resources. We show
  that pretraining on a carefully selected, task-relevant dataset can match or
  even surpass large-scale pretraining, while using as little as 1/24th of the
  computational cost. We introduce the Chemical Similarity Index (CSI), a novel
  metric inspired by computer vision's Fr\'echet Inception Distance, for
  molecular graphs which quantifies the alignment between upstream pretraining
  datasets and downstream tasks. By selecting the most relevant dataset with
  minimal CSI distance, we show that models pretrained on a smaller, focused
  dataset consistently outperform those pretrained on massive, mixed datasets
  such as JMP, even when those larger datasets include the relevant dataset.
  Counterintuitively, we also find that indiscriminately adding more data can
  degrade model performance when the additional data poorly aligns with the task
  at hand. Our findings highlight that quality often outperforms quantity in
  pretraining for atomic property prediction.