title: GeoGalactica: A Scientific Large Language Model in Geoscience
summary: Large language models (LLMs) have achieved huge success for their general
  knowledge and ability to solve a wide spectrum of tasks in natural language
  processing (NLP). Due to their impressive abilities, LLMs have shed light on
  potential inter-discipline applications to foster scientific discoveries of a
  specific domain by using artificial intelligence (AI for science, AI4S). In the
  meantime, utilizing NLP techniques in geoscience research and practice is wide
  and convoluted, contributing from knowledge extraction and document
  classification to question answering and knowledge discovery. In this work, we
  take the initial step to leverage LLM for science, through a rather
  straightforward approach. We try to specialize an LLM into geoscience, by
  further pre-training the model with a vast amount of texts in geoscience, as
  well as supervised fine-tuning (SFT) the resulting model with our custom
  collected instruction tuning dataset. These efforts result in a model
  GeoGalactica consisting of 30 billion parameters. To our best knowledge, it is
  the largest language model for the geoscience domain. More specifically,
  GeoGalactica is from further pre-training of Galactica. We train GeoGalactica
  over a geoscience-related text corpus containing 65 billion tokens curated from
  extensive data sources in the big science project Deep-time Digital Earth
  (DDE), preserving as the largest geoscience-specific text corpus. Then we
  fine-tune the model with 1 million pairs of instruction-tuning data consisting
  of questions that demand professional geoscience knowledge to answer. In this
  technical report, we will illustrate in detail all aspects of GeoGalactica,
  including data collection, data cleaning, base model selection, pre-training,
  SFT, and evaluation. We open-source our data curation tools and the checkpoints
  of GeoGalactica during the first 3/4 of pre-training.
paper_authors._id: 65938c64bec7ad2647d20430
paper_authors.name: Zhouhan Lin
paper_authors._id: 65938c64bec7ad2647d20431
paper_authors.name: Cheng Deng
paper_authors._id: 65938c64bec7ad2647d20432
paper_authors.name: Le Zhou
paper_authors._id: 65938c64bec7ad2647d20433
paper_authors.name: Tianhang Zhang
paper_authors._id: 65938c64bec7ad2647d20434
paper_authors.name: Yi Xu
paper_authors._id: 65938c64bec7ad2647d20435
paper_authors.name: Yutong Xu
paper_authors._id: 65938c64bec7ad2647d20436
paper_authors.name: Zhongmou He
paper_authors._id: 65938c64bec7ad2647d20437
paper_authors.name: Yuanyuan Shi
paper_authors._id: 65938c64bec7ad2647d20438
paper_authors.name: Beiya Dai
paper_authors._id: 65938c64bec7ad2647d20439
paper_authors.name: Yunchong Song
paper_authors._id: 65938c64bec7ad2647d2043a
paper_authors.name: Boyi Zeng
paper_authors._id: 65938c64bec7ad2647d2043b
paper_authors.name: Qiyuan Chen
paper_authors._id: 65938c64bec7ad2647d2043c
paper_authors.name: Tao Shi
paper_authors._id: 65938c64bec7ad2647d2043d
paper_authors.name: Tianyu Huang
paper_authors._id: 65938c64bec7ad2647d2043e
paper_authors.name: Yiwei Xu
paper_authors._id: 65938c64bec7ad2647d2043f
paper_authors.name: Shu Wang
paper_authors._id: 65938c64bec7ad2647d20440
paper_authors.name: Luoyi Fu
paper_authors._id: 65938c64bec7ad2647d20441
paper_authors.name: Weinan Zhang
paper_authors._id: 65938c64bec7ad2647d20442
paper_authors.name: Junxian He
paper_authors._id: 65938c64bec7ad2647d20443
paper_authors.name: Chao Ma
paper_authors._id: 65938c64bec7ad2647d20444
paper_authors.name: Yunqiang Zhu
paper_authors._id: 65938c64bec7ad2647d20445
paper_authors.name: Xinbing Wang
paper_authors._id: 65938c64bec7ad2647d20446
paper_authors.name: Chenghu Zhou
paper_title: GeoGalactica: A Scientific Large Language Model in Geoscience
paper_summary: Large language models (LLMs) have achieved huge success for their general
  knowledge and ability to solve a wide spectrum of tasks in natural language
  processing (NLP). Due to their impressive abilities, LLMs have shed light on
  potential inter-discipline applications to foster scientific discoveries of a
  specific domain by using artificial intelligence (AI for science, AI4S). In the
  meantime, utilizing NLP techniques in geoscience research and practice is wide
  and convoluted, contributing from knowledge extraction and document
  classification to question answering and knowledge discovery. In this work, we
  take the initial step to leverage LLM for science, through a rather
  straightforward approach. We try to specialize an LLM into geoscience, by
  further pre-training the model with a vast amount of texts in geoscience, as
  well as supervised fine-tuning (SFT) the resulting model with our custom
  collected instruction tuning dataset. These efforts result in a model
  GeoGalactica consisting of 30 billion parameters. To our best knowledge, it is
  the largest language model for the geoscience domain. More specifically,
  GeoGalactica is from further pre-training of Galactica. We train GeoGalactica
  over a geoscience-related text corpus containing 65 billion tokens curated from
  extensive data sources in the big science project Deep-time Digital Earth
  (DDE), preserving as the largest geoscience-specific text corpus. Then we
  fine-tune the model with 1 million pairs of instruction-tuning data consisting
  of questions that demand professional geoscience knowledge to answer. In this
  technical report, we will illustrate in detail all aspects of GeoGalactica,
  including data collection, data cleaning, base model selection, pre-training,
  SFT, and evaluation. We open-source our data curation tools and the checkpoints
  of GeoGalactica during the first 3/4 of pre-training.
paper_ai_keywords: large language models (LLMs) natural language processing (NLP) artificial intelligence (AI for science, AI4S) knowledge extraction document classification question answering knowledge discovery specialized LLM further pre-training supervised fine-tuning (SFT) instruction tuning dataset GeoGalactica Galactica geoscience-related text corpus Deep-time Digital Earth (DDE) text tokens instruction-tuning data