GeoGalactica: A Scientific Large Language Model in Geoscience
Large language models (LLMs) have achieved huge success for their general
knowledge and ability to solve a wide spectrum of tasks in natural language
processing (NLP). Due to their impressive abilities, LLMs have shed light on
potential inter-discipline applications to foster scientific discoveries of a
specific domain by using artificial intelligence (AI for science, AI4S). In the
meantime, utilizing NLP techniques in geoscience research and practice is wide
and convoluted, contributing from knowledge extraction and document
classification to question answering and knowledge discovery. In this work, we
take the initial step to leverage LLM for science, through a rather
straightforward approach. We try to specialize an LLM into geoscience, by
further pre-training the model with a vast amount of texts in geoscience, as
well as supervised fine-tuning (SFT) the resulting model with our custom
collected instruction tuning dataset. These efforts result in a model
GeoGalactica consisting of 30 billion parameters. To our best knowledge, it is
the largest language model for the geoscience domain. More specifically,
GeoGalactica is from further pre-training of Galactica. We train GeoGalactica
over a geoscience-related text corpus containing 65 billion tokens curated from
extensive data sources in the big science project Deep-time Digital Earth
(DDE), preserving as the largest geoscience-specific text corpus. Then we
fine-tune the model with 1 million pairs of instruction-tuning data consisting
of questions that demand professional geoscience knowledge to answer. In this
technical report, we will illustrate in detail all aspects of GeoGalactica,
including data collection, data cleaning, base model selection, pre-training,
SFT, and evaluation. We open-source our data curation tools and the checkpoints
of GeoGalactica during the first 3/4 of pre-training.
65938c64bec7ad2647d20430
Zhouhan Lin
65938c64bec7ad2647d20431
Cheng Deng
65938c64bec7ad2647d20432
Le Zhou
65938c64bec7ad2647d20433
Tianhang Zhang
65938c64bec7ad2647d20434
Yi Xu
65938c64bec7ad2647d20435
Yutong Xu
65938c64bec7ad2647d20436
Zhongmou He
65938c64bec7ad2647d20437
Yuanyuan Shi
65938c64bec7ad2647d20438
Beiya Dai
65938c64bec7ad2647d20439
Yunchong Song
65938c64bec7ad2647d2043a
Boyi Zeng
65938c64bec7ad2647d2043b
Qiyuan Chen
65938c64bec7ad2647d2043c
Tao Shi
65938c64bec7ad2647d2043d
Tianyu Huang
65938c64bec7ad2647d2043e
Yiwei Xu
65938c64bec7ad2647d2043f
Shu Wang
65938c64bec7ad2647d20440
Luoyi Fu
65938c64bec7ad2647d20441
Weinan Zhang
65938c64bec7ad2647d20442
Junxian He
65938c64bec7ad2647d20443
Chao Ma
65938c64bec7ad2647d20444
Yunqiang Zhu
65938c64bec7ad2647d20445
Xinbing Wang
65938c64bec7ad2647d20446
Chenghu Zhou
GeoGalactica: A Scientific Large Language Model in Geoscience
Large language models (LLMs) have achieved huge success for their general
knowledge and ability to solve a wide spectrum of tasks in natural language
processing (NLP). Due to their impressive abilities, LLMs have shed light on
potential inter-discipline applications to foster scientific discoveries of a
specific domain by using artificial intelligence (AI for science, AI4S). In the
meantime, utilizing NLP techniques in geoscience research and practice is wide
and convoluted, contributing from knowledge extraction and document
classification to question answering and knowledge discovery. In this work, we
take the initial step to leverage LLM for science, through a rather
straightforward approach. We try to specialize an LLM into geoscience, by
further pre-training the model with a vast amount of texts in geoscience, as
well as supervised fine-tuning (SFT) the resulting model with our custom
collected instruction tuning dataset. These efforts result in a model
GeoGalactica consisting of 30 billion parameters. To our best knowledge, it is
the largest language model for the geoscience domain. More specifically,
GeoGalactica is from further pre-training of Galactica. We train GeoGalactica
over a geoscience-related text corpus containing 65 billion tokens curated from
extensive data sources in the big science project Deep-time Digital Earth
(DDE), preserving as the largest geoscience-specific text corpus. Then we
fine-tune the model with 1 million pairs of instruction-tuning data consisting
of questions that demand professional geoscience knowledge to answer. In this
technical report, we will illustrate in detail all aspects of GeoGalactica,
including data collection, data cleaning, base model selection, pre-training,
SFT, and evaluation. We open-source our data curation tools and the checkpoints
of GeoGalactica during the first 3/4 of pre-training.
GeoGalactica, a specialized LLM for geoscience with 30 billion parameters, was created through further pre-training and supervised fine-tuning with a large geoscience-related text corpus and professional instruction-tuning data.
LLMs NLP AI4S knowledge extraction document classification question answering knowledge discovery supervised fine-tuning SFT pre-training instruction tuning dataset geoscience-specific text corpus Deep-time Digital Earth DDE evaluation data curation tools