title: INDUS: Effective and Efficient Language Models for Scientific
    Applications
summary: Large language models (LLMs) trained on general domain corpora showed
  remarkable results on natural language processing (NLP) tasks. However,
  previous research demonstrated LLMs trained using domain-focused corpora
  perform better on specialized tasks. Inspired by this pivotal insight, we
  developed INDUS, a comprehensive suite of LLMs tailored for the Earth science,
  biology, physics, heliophysics, planetary sciences and astrophysics domains and
  trained using curated scientific corpora drawn from diverse data sources. The
  suite of models include: (1) an encoder model trained using domain-specific
  vocabulary and corpora to address natural language understanding tasks, (2) a
  contrastive-learning-based general text embedding model trained using a diverse
  set of datasets drawn from multiple sources to address information retrieval
  tasks and (3) smaller versions of these models created using knowledge
  distillation techniques to address applications which have latency or resource
  constraints. We also created three new scientific benchmark datasets namely,
  CLIMATE-CHANGE-NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR
  (IR) to accelerate research in these multi-disciplinary fields. Finally, we
  show that our models outperform both general-purpose encoders (RoBERTa) and
  existing domain-specific encoders (SciBERT) on these new tasks as well as
  existing benchmark tasks in the domains of interest.
paper_authors._id: 664b68b7bfd9b93ba4ac0026
paper_authors.name: Bishwaranjan Bhattacharjee
paper_authors._id: 664b68b7bfd9b93ba4ac0027
paper_authors.name: Aashka Trivedi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:23:50.141Z
paper_authors._id: 6620044150fab5fc315c5457
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6620044150fab5fc315c5457/WaUpGHLpS0BWY2Cjr5K4S.jpeg
paper_authors.fullname: Aashka Trivedi
paper_authors.type: user
paper_authors.user: aashkaaa
paper_authors._id: 664b68b7bfd9b93ba4ac0028
paper_authors.name: Masayasu Muraoka
paper_authors._id: 664b68b7bfd9b93ba4ac0029
paper_authors.name: Muthukumaran Ramasubramanian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T09:01:27.127Z
paper_authors._id: 61099e5d86580d4580767226
paper_authors.avatarUrl: /avatars/76ab955b5388c8bd732a19c0516d72c5.svg
paper_authors.fullname: Ramasubramanian
paper_authors.type: user
paper_authors.user: Muthukumaran
paper_authors._id: 664b68b7bfd9b93ba4ac002a
paper_authors.name: Takuma Udagawa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:24:38.671Z
paper_authors._id: 652f684161ce812084940d19
paper_authors.avatarUrl: /avatars/21a233344781e6a6f60b17a7f38f006c.svg
paper_authors.fullname: Takuma Udagawa
paper_authors.type: user
paper_authors.user: uddie
paper_authors._id: 664b68b7bfd9b93ba4ac002b
paper_authors.name: Iksha Gurung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:24:45.485Z
paper_authors._id: 6499a2d9b9a0112f18e15a09
paper_authors.avatarUrl: /avatars/1576e485309c45cbf1e574354667670e.svg
paper_authors.fullname: Iksha Gurung
paper_authors.type: user
paper_authors.user: xhagrg
paper_authors._id: 664b68b7bfd9b93ba4ac002c
paper_authors.name: Rong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:24:54.102Z
paper_authors._id: 62bc74b71cb22b7f9bb17e45
paper_authors.avatarUrl: /avatars/5d2e9f98ee36c8ebc68911a8c2369646.svg
paper_authors.fullname: Rong Zhang
paper_authors.type: user
paper_authors.user: rongzhangibm
paper_authors._id: 664b68b7bfd9b93ba4ac002d
paper_authors.name: Bharath Dandala
paper_authors._id: 664b68b7bfd9b93ba4ac002e
paper_authors.name: Rahul Ramachandran
paper_authors._id: 664b68b7bfd9b93ba4ac002f
paper_authors.name: Manil Maskey
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:25:29.849Z
paper_authors._id: 64ba94321e53e10faf0c14ca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ba94321e53e10faf0c14ca/leAHIQoy8nk74Mdut3MJ7.jpeg
paper_authors.fullname: Manil Maskey
paper_authors.type: user
paper_authors.user: manilmaskey
paper_authors._id: 664b68b7bfd9b93ba4ac0030
paper_authors.name: Kayleen Bugbee
paper_authors._id: 664b68b7bfd9b93ba4ac0031
paper_authors.name: Mike Little
paper_authors._id: 664b68b7bfd9b93ba4ac0032
paper_authors.name: Elizabeth Fancher
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:25:55.994Z
paper_authors._id: 64bec6b3b1a6188809b71efd
paper_authors.avatarUrl: /avatars/a42e4aa0f014287c23d0445f3459e4b2.svg
paper_authors.fullname: Elizabeth Fancher
paper_authors.type: user
paper_authors.user: efancher
paper_authors._id: 664b68b7bfd9b93ba4ac0033
paper_authors.name: Lauren Sanders
paper_authors._id: 664b68b7bfd9b93ba4ac0034
paper_authors.name: Sylvain Costes
paper_authors._id: 664b68b7bfd9b93ba4ac0035
paper_authors.name: Sergi Blanco-Cuaresma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:26:24.255Z
paper_authors._id: 627a6840f652717251c77628
paper_authors.avatarUrl: /avatars/0be9a72c5efe835bbfbc7730d819dede.svg
paper_authors.fullname: Sergi Blanco-Cuaresma
paper_authors.type: user
paper_authors.user: marblestation
paper_authors._id: 664b68b7bfd9b93ba4ac0036
paper_authors.name: Kelly Lockhart
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:26:30.619Z
paper_authors._id: 646258c3d9b49021880f724e
paper_authors.avatarUrl: /avatars/dfefd298a9973328671a3dca6e2cb04a.svg
paper_authors.fullname: Kelly Lockhart
paper_authors.type: user
paper_authors.user: kelockhart
paper_authors._id: 664b68b7bfd9b93ba4ac0037
paper_authors.name: Thomas Allen
paper_authors._id: 664b68b7bfd9b93ba4ac0038
paper_authors.name: Felix Grazes
paper_authors._id: 664b68b7bfd9b93ba4ac0039
paper_authors.name: Megan Ansdel
paper_authors._id: 664b68b7bfd9b93ba4ac003a
paper_authors.name: Alberto Accomazzi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:27:07.546Z
paper_authors._id: 65083c0018f9c580cde6494c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65083c0018f9c580cde6494c/W9sBT0xvexHcwDkTaxpgA.jpeg
paper_authors.fullname: Alberto Accomazzi
paper_authors.type: user
paper_authors.user: aaccomazzi
paper_authors._id: 664b68b7bfd9b93ba4ac003b
paper_authors.name: Yousef El-Kurdi
paper_authors._id: 664b68b7bfd9b93ba4ac003c
paper_authors.name: Davis Wertheimer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:27:22.285Z
paper_authors._id: 6614072e2938f840baacfa11
paper_authors.avatarUrl: /avatars/894fc389c32b13a6f6a894dab4fdd65b.svg
paper_authors.fullname: Davis Wertheimer
paper_authors.type: user
paper_authors.user: daviswer
paper_authors._id: 664b68b7bfd9b93ba4ac003d
paper_authors.name: Birgit Pfitzmann
paper_authors._id: 664b68b7bfd9b93ba4ac003e
paper_authors.name: Cesar Berrospi Ramis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:27:33.952Z
paper_authors._id: 661e328c271f493e58b1f2ad
paper_authors.avatarUrl: /avatars/0146ffdad7565b28687645c1dbdd4ea7.svg
paper_authors.fullname: Cesar Berrospi Ramis
paper_authors.type: user
paper_authors.user: ceberam
paper_authors._id: 664b68b7bfd9b93ba4ac003f
paper_authors.name: Michele Dolfi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:27:41.816Z
paper_authors._id: 63c64dd877caf00391004e20
paper_authors.avatarUrl: /avatars/6350443da6405ea7beccf47a80d8710e.svg
paper_authors.fullname: Michele Dolfi
paper_authors.type: user
paper_authors.user: dolfim-ibm
paper_authors._id: 664b68b7bfd9b93ba4ac0040
paper_authors.name: Rafael Teixeira de Lima
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:27:48.919Z
paper_authors._id: 65ce0dbef3bbc55ca4d0528f
paper_authors.avatarUrl: /avatars/ae19424457dbec49579fe5a4ae7177b0.svg
paper_authors.fullname: Rafael Teixeira de Lima
paper_authors.type: user
paper_authors.user: rtdl
paper_authors._id: 664b68b7bfd9b93ba4ac0041
paper_authors.name: Panos Vegenas
paper_authors._id: 664b68b7bfd9b93ba4ac0042
paper_authors.name: S. Karthik Mukkavilli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:27:59.672Z
paper_authors._id: 64cd0861b16aabccbe34553b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64cd0861b16aabccbe34553b/T-IAqBIG6SzXnUOdagAii.jpeg
paper_authors.fullname: S. Karthik Mukkavilli
paper_authors.type: user
paper_authors.user: drmukkavilli
paper_authors._id: 664b68b7bfd9b93ba4ac0043
paper_authors.name: Peter Staar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:28:06.674Z
paper_authors._id: 63c6522d8fddd19431f48bcc
paper_authors.avatarUrl: /avatars/e2ec9c76656b7520472718595f9dafe1.svg
paper_authors.fullname: Peter W. J. Staar
paper_authors.type: user
paper_authors.user: PeterStaar
paper_authors._id: 664b68b7bfd9b93ba4ac0044
paper_authors.name: Sanaz Vahidinia
paper_authors._id: 664b68b7bfd9b93ba4ac0045
paper_authors.name: Ryan McGranaghan
paper_authors._id: 664b68b7bfd9b93ba4ac0046
paper_authors.name: Armin Mehrabian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:28:22.329Z
paper_authors._id: 62d0696a485856cd710e549b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d0696a485856cd710e549b/HaOWjdJyVImULLykvI_ez.jpeg
paper_authors.fullname: Armin Mehrabian
paper_authors.type: user
paper_authors.user: arminmehrabian
paper_authors._id: 664b68b7bfd9b93ba4ac0047
paper_authors.name: Tsendgar Lee
paper_title: INDUS: Effective and Efficient Language Models for Scientific
    Applications
paper_summary: Large language models (LLMs) trained on general domain corpora showed
  remarkable results on natural language processing (NLP) tasks. However,
  previous research demonstrated LLMs trained using domain-focused corpora
  perform better on specialized tasks. Inspired by this pivotal insight, we
  developed INDUS, a comprehensive suite of LLMs tailored for the Earth science,
  biology, physics, heliophysics, planetary sciences and astrophysics domains and
  trained using curated scientific corpora drawn from diverse data sources. The
  suite of models include: (1) an encoder model trained using domain-specific
  vocabulary and corpora to address natural language understanding tasks, (2) a
  contrastive-learning-based general text embedding model trained using a diverse
  set of datasets drawn from multiple sources to address information retrieval
  tasks and (3) smaller versions of these models created using knowledge
  distillation techniques to address applications which have latency or resource
  constraints. We also created three new scientific benchmark datasets namely,
  CLIMATE-CHANGE-NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR
  (IR) to accelerate research in these multi-disciplinary fields. Finally, we
  show that our models outperform both general-purpose encoders (RoBERTa) and
  existing domain-specific encoders (SciBERT) on these new tasks as well as
  existing benchmark tasks in the domains of interest.
paper_ai_summary: INDUS, a suite of domain-specific LLMs trained on curated scientific corpora, surpasses general-purpose and existing domain-specific models on specialized NLP tasks and new scientific benchmarks.
paper_ai_keywords: LLMs natural language processing domain-specific models encoder model contrastive-learning-based general text embedding model knowledge distillation CLIMATE-CHANGE-NER NASA-QA NASA-IR RoBERTa SciBERT