title: TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation
summary: In this paper, we propose Text-based Open Molecule Generation Benchmark
  (TOMG-Bench), the first benchmark to evaluate the open-domain molecule
  generation capability of LLMs. TOMG-Bench encompasses a dataset of three major
  tasks: molecule editing (MolEdit), molecule optimization (MolOpt), and
  customized molecule generation (MolCustom). Each task further contains three
  subtasks, with each subtask comprising 5,000 test samples. Given the inherent
  complexity of open molecule generation, we have also developed an automated
  evaluation system that helps measure both the quality and the accuracy of the
  generated molecules. Our comprehensive benchmarking of 25 LLMs reveals the
  current limitations and potential areas for improvement in text-guided molecule
  discovery. Furthermore, with the assistance of OpenMolIns, a specialized
  instruction tuning dataset proposed for solving challenges raised by
  TOMG-Bench, Llama3.1-8B could outperform all the open-source general LLMs, even
  surpassing GPT-3.5-turbo by 46.5\% on TOMG-Bench. Our codes and datasets are
  available through https://github.com/phenixace/TOMG-Bench.
paper_authors._id: 6764de0ca246952fabef4309
paper_authors.name: Jiatong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:36:31.168Z
paper_authors._id: 6438e55cb2ea24b52ebc45ec
paper_authors.avatarUrl: /avatars/e13c7398f77e7e0bd5eed03102aa5c36.svg
paper_authors.fullname: Jiatong LI
paper_authors.type: user
paper_authors.user: phenixace
paper_authors._id: 6764de0ca246952fabef430a
paper_authors.name: Junxian Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:32:48.973Z
paper_authors._id: 656ae4088fb1ddf0d5ec9ac5
paper_authors.avatarUrl: /avatars/e38468d2c0274f3c0f5732f30a2e3436.svg
paper_authors.fullname: Junxian Li
paper_authors.type: user
paper_authors.user: Duke-de-Artois
paper_authors._id: 6764de0ca246952fabef430b
paper_authors.name: Yunqing Liu
paper_authors._id: 6764de0ca246952fabef430c
paper_authors.name: Dongzhan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:30:10.820Z
paper_authors._id: 6538b861613fe158bd581e35
paper_authors.avatarUrl: /avatars/6817dbfe903675721fd227058b0a91ac.svg
paper_authors.fullname: Dongzhan Zhou
paper_authors.type: user
paper_authors.user: schrodingers-tiger
paper_authors._id: 6764de0ca246952fabef430d
paper_authors.name: Qing Li
paper_title: TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation
paper_summary: In this paper, we propose Text-based Open Molecule Generation Benchmark
  (TOMG-Bench), the first benchmark to evaluate the open-domain molecule
  generation capability of LLMs. TOMG-Bench encompasses a dataset of three major
  tasks: molecule editing (MolEdit), molecule optimization (MolOpt), and
  customized molecule generation (MolCustom). Each task further contains three
  subtasks, with each subtask comprising 5,000 test samples. Given the inherent
  complexity of open molecule generation, we have also developed an automated
  evaluation system that helps measure both the quality and the accuracy of the
  generated molecules. Our comprehensive benchmarking of 25 LLMs reveals the
  current limitations and potential areas for improvement in text-guided molecule
  discovery. Furthermore, with the assistance of OpenMolIns, a specialized
  instruction tuning dataset proposed for solving challenges raised by
  TOMG-Bench, Llama3.1-8B could outperform all the open-source general LLMs, even
  surpassing GPT-3.5-turbo by 46.5\% on TOMG-Bench. Our codes and datasets are
  available through https://github.com/phenixace/TOMG-Bench.
paper_ai_keywords: text-based open molecule generation TOMG-Bench molecule editing (MolEdit) molecule optimization (MolOpt) customized molecule generation (MolCustom) text-guided molecule discovery OpenMolIns Llama3.1-8B GPT-3.5-turbo