SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding
Scientific literature understanding is crucial for extracting targeted
information and garnering insights, thereby significantly advancing scientific
discovery. Despite the remarkable success of Large Language Models (LLMs), they
face challenges in scientific literature understanding, primarily due to (1) a
lack of scientific knowledge and (2) unfamiliarity with specialized scientific
tasks.
  To develop an LLM specialized in scientific literature understanding, we
propose a hybrid strategy that integrates continual pre-training (CPT) and
supervised fine-tuning (SFT), to simultaneously infuse scientific domain
knowledge and enhance instruction-following capabilities for domain-specific
tasks.cIn this process, we identify two key challenges: (1) constructing
high-quality CPT corpora, and (2) generating diverse SFT instructions. We
address these challenges through a meticulous pipeline, including PDF text
extraction, parsing content error correction, quality filtering, and synthetic
instruction creation. Applying this strategy, we present a suite of LLMs:
SciLitLLM, specialized in scientific literature understanding. These models
demonstrate promising performance on scientific literature understanding
benchmarks.
  Our contributions are threefold: (1) We present an effective framework that
integrates CPT and SFT to adapt LLMs to scientific literature understanding,
which can also be easily adapted to other domains. (2) We propose an LLM-based
synthesis method to generate diverse and high-quality scientific instructions,
resulting in a new instruction set -- SciLitIns -- for supervised fine-tuning
in less-represented scientific domains. (3) SciLitLLM achieves promising
performance improvements on scientific literature understanding benchmarks.
66d57d30c75e3c024260811c
Sihang Li
extracted_pending
2024-09-02T08:54:10.133Z
635913a68f38318bfabc90ea
https://cdn-avatars.huggingface.co/v1/production/uploads/635913a68f38318bfabc90ea/29M50hPy2nMwMjMElAEgA.jpeg
Sihang Li
user
Sihangli
66d57d30c75e3c024260811d
Jin Huang
66d57d30c75e3c024260811e
Jiaxi Zhuang
admin_assigned
2024-09-02T09:32:26.844Z
6673dc342aadd5db0b61e562
/avatars/0a1c5fa4d2edac46e68dc6d6115dbe77.svg
jiaxi zhuang
user
Jesse-zjx
66d57d30c75e3c024260811f
Yaorui Shi
admin_assigned
2024-09-02T09:32:32.846Z
63edd2d1f765928ceeb49057
https://cdn-avatars.huggingface.co/v1/production/uploads/1676530369930-noauth.png
Yaorui SHI
user
yrshi
66d57d30c75e3c0242608120
Xiaochen Cai
admin_assigned
2024-09-02T09:33:08.817Z
64ac17a15769f5f88bbf5088
/avatars/f8f50390cf767cc62ae612e6b600a340.svg
caixiaochen
user
caixc97
66d57d30c75e3c0242608121
Mingjun Xu
admin_assigned
2024-09-02T09:33:15.181Z
653b75e286b88947d5d64eff
/avatars/259e5bd353fdb05d2bd102b6d5c2d589.svg
Mingjun Xu
user
xumingjun
66d57d30c75e3c0242608122
Xiang Wang
extracted_confirmed
2024-10-14T12:32:20.684Z
65fca775fa59bdf4737b1a84
/avatars/a161b510bde8f57e7686cbb0b4aa6a52.svg
Xiang Wang
user
xiangwang1223
66d57d30c75e3c0242608123
Linfeng Zhang
66d57d30c75e3c0242608124
Guolin Ke
admin_assigned
2024-09-02T09:33:40.436Z
6348de0c62c668c7b48d83c9
/avatars/7296ea9bb301e19c10926022959b2023.svg
Guolin Ke
user
guolinke
66d57d30c75e3c0242608125
Hengxing Cai
admin_assigned
2024-09-02T09:33:46.432Z
6340c214181f5648a58f5f82
/avatars/2ee353d8ec2b0ebc5b58d0641beddc26.svg
Hengxing Cai
user
enjoy
SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding
Scientific literature understanding is crucial for extracting targeted
information and garnering insights, thereby significantly advancing scientific
discovery. Despite the remarkable success of Large Language Models (LLMs), they
face challenges in scientific literature understanding, primarily due to (1) a
lack of scientific knowledge and (2) unfamiliarity with specialized scientific
tasks.
  To develop an LLM specialized in scientific literature understanding, we
propose a hybrid strategy that integrates continual pre-training (CPT) and
supervised fine-tuning (SFT), to simultaneously infuse scientific domain
knowledge and enhance instruction-following capabilities for domain-specific
tasks.cIn this process, we identify two key challenges: (1) constructing
high-quality CPT corpora, and (2) generating diverse SFT instructions. We
address these challenges through a meticulous pipeline, including PDF text
extraction, parsing content error correction, quality filtering, and synthetic
instruction creation. Applying this strategy, we present a suite of LLMs:
SciLitLLM, specialized in scientific literature understanding. These models
demonstrate promising performance on scientific literature understanding
benchmarks.
  Our contributions are threefold: (1) We present an effective framework that
integrates CPT and SFT to adapt LLMs to scientific literature understanding,
which can also be easily adapted to other domains. (2) We propose an LLM-based
synthesis method to generate diverse and high-quality scientific instructions,
resulting in a new instruction set -- SciLitIns -- for supervised fine-tuning
in less-represented scientific domains. (3) SciLitLLM achieves promising
performance improvements on scientific literature understanding benchmarks.
A hybrid approach combining continual pre-training and supervised fine-tuning creates specialized LLMs for scientific literature, addressing challenges in constructing corpora and generating instructions.
Large Language Models LLMs continual pre-training CPT supervised fine-tuning SFT scientific literature understanding domain-specific tasks PDF text extraction content parsing error correction quality filtering synthetic instruction creation SciLitLLM scientific literature understanding benchmarks SciLitIns