title: SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding
summary: Scientific literature understanding is crucial for extracting targeted
  information and garnering insights, thereby significantly advancing scientific
  discovery. Despite the remarkable success of Large Language Models (LLMs), they
  face challenges in scientific literature understanding, primarily due to (1) a
  lack of scientific knowledge and (2) unfamiliarity with specialized scientific
  tasks.
    To develop an LLM specialized in scientific literature understanding, we
  propose a hybrid strategy that integrates continual pre-training (CPT) and
  supervised fine-tuning (SFT), to simultaneously infuse scientific domain
  knowledge and enhance instruction-following capabilities for domain-specific
  tasks.cIn this process, we identify two key challenges: (1) constructing
  high-quality CPT corpora, and (2) generating diverse SFT instructions. We
  address these challenges through a meticulous pipeline, including PDF text
  extraction, parsing content error correction, quality filtering, and synthetic
  instruction creation. Applying this strategy, we present a suite of LLMs:
  SciLitLLM, specialized in scientific literature understanding. These models
  demonstrate promising performance on scientific literature understanding
  benchmarks.
    Our contributions are threefold: (1) We present an effective framework that
  integrates CPT and SFT to adapt LLMs to scientific literature understanding,
  which can also be easily adapted to other domains. (2) We propose an LLM-based
  synthesis method to generate diverse and high-quality scientific instructions,
  resulting in a new instruction set -- SciLitIns -- for supervised fine-tuning
  in less-represented scientific domains. (3) SciLitLLM achieves promising
  performance improvements on scientific literature understanding benchmarks.
paper_authors._id: 66d57d30c75e3c024260811c
paper_authors.name: Sihang Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-09-02T08:54:10.133Z
paper_authors._id: 635913a68f38318bfabc90ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635913a68f38318bfabc90ea/29M50hPy2nMwMjMElAEgA.jpeg
paper_authors.fullname: Sihang Li
paper_authors.type: user
paper_authors.user: Sihangli
paper_authors._id: 66d57d30c75e3c024260811d
paper_authors.name: Jin Huang
paper_authors._id: 66d57d30c75e3c024260811e
paper_authors.name: Jiaxi Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:32:26.844Z
paper_authors._id: 6673dc342aadd5db0b61e562
paper_authors.avatarUrl: /avatars/0a1c5fa4d2edac46e68dc6d6115dbe77.svg
paper_authors.fullname: jiaxi zhuang
paper_authors.type: user
paper_authors.user: Jesse-zjx
paper_authors._id: 66d57d30c75e3c024260811f
paper_authors.name: Yaorui Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:32:32.846Z
paper_authors._id: 63edd2d1f765928ceeb49057
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676530369930-noauth.png
paper_authors.fullname: Yaorui SHI
paper_authors.type: user
paper_authors.user: yrshi
paper_authors._id: 66d57d30c75e3c0242608120
paper_authors.name: Xiaochen Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:33:08.817Z
paper_authors._id: 64ac17a15769f5f88bbf5088
paper_authors.avatarUrl: /avatars/f8f50390cf767cc62ae612e6b600a340.svg
paper_authors.fullname: caixiaochen
paper_authors.type: user
paper_authors.user: caixc97
paper_authors._id: 66d57d30c75e3c0242608121
paper_authors.name: Mingjun Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:33:15.181Z
paper_authors._id: 653b75e286b88947d5d64eff
paper_authors.avatarUrl: /avatars/259e5bd353fdb05d2bd102b6d5c2d589.svg
paper_authors.fullname: Mingjun Xu
paper_authors.type: user
paper_authors.user: xumingjun
paper_authors._id: 66d57d30c75e3c0242608122
paper_authors.name: Xiang Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-14T12:32:20.684Z
paper_authors._id: 65fca775fa59bdf4737b1a84
paper_authors.avatarUrl: /avatars/a161b510bde8f57e7686cbb0b4aa6a52.svg
paper_authors.fullname: Xiang Wang
paper_authors.type: user
paper_authors.user: xiangwang1223
paper_authors._id: 66d57d30c75e3c0242608123
paper_authors.name: Linfeng Zhang
paper_authors._id: 66d57d30c75e3c0242608124
paper_authors.name: Guolin Ke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:33:40.436Z
paper_authors._id: 6348de0c62c668c7b48d83c9
paper_authors.avatarUrl: /avatars/7296ea9bb301e19c10926022959b2023.svg
paper_authors.fullname: Guolin Ke
paper_authors.type: user
paper_authors.user: guolinke
paper_authors._id: 66d57d30c75e3c0242608125
paper_authors.name: Hengxing Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-02T09:33:46.432Z
paper_authors._id: 6340c214181f5648a58f5f82
paper_authors.avatarUrl: /avatars/2ee353d8ec2b0ebc5b58d0641beddc26.svg
paper_authors.fullname: Hengxing Cai
paper_authors.type: user
paper_authors.user: enjoy
paper_title: SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding
paper_summary: Scientific literature understanding is crucial for extracting targeted
  information and garnering insights, thereby significantly advancing scientific
  discovery. Despite the remarkable success of Large Language Models (LLMs), they
  face challenges in scientific literature understanding, primarily due to (1) a
  lack of scientific knowledge and (2) unfamiliarity with specialized scientific
  tasks.
    To develop an LLM specialized in scientific literature understanding, we
  propose a hybrid strategy that integrates continual pre-training (CPT) and
  supervised fine-tuning (SFT), to simultaneously infuse scientific domain
  knowledge and enhance instruction-following capabilities for domain-specific
  tasks.cIn this process, we identify two key challenges: (1) constructing
  high-quality CPT corpora, and (2) generating diverse SFT instructions. We
  address these challenges through a meticulous pipeline, including PDF text
  extraction, parsing content error correction, quality filtering, and synthetic
  instruction creation. Applying this strategy, we present a suite of LLMs:
  SciLitLLM, specialized in scientific literature understanding. These models
  demonstrate promising performance on scientific literature understanding
  benchmarks.
    Our contributions are threefold: (1) We present an effective framework that
  integrates CPT and SFT to adapt LLMs to scientific literature understanding,
  which can also be easily adapted to other domains. (2) We propose an LLM-based
  synthesis method to generate diverse and high-quality scientific instructions,
  resulting in a new instruction set -- SciLitIns -- for supervised fine-tuning
  in less-represented scientific domains. (3) SciLitLLM achieves promising
  performance improvements on scientific literature understanding benchmarks.
paper_ai_keywords: Large Language Models (LLMs) continual pre-training (CPT) supervised fine-tuning (SFT) domain-specific tasks PDF text extraction synthetic instruction creation SciLitLLM scientific literature understanding benchmarks SciLitIns