title: OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented
    LMs
summary: Scientific progress depends on researchers' ability to synthesize the growing
  body of literature. Can large language models (LMs) assist scientists in this
  task? We introduce OpenScholar, a specialized retrieval-augmented LM that
  answers scientific queries by identifying relevant passages from 45 million
  open-access papers and synthesizing citation-backed responses. To evaluate
  OpenScholar, we develop ScholarQABench, the first large-scale multi-domain
  benchmark for literature search, comprising 2,967 expert-written queries and
  208 long-form answers across computer science, physics, neuroscience, and
  biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and
  PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o
  hallucinates citations 78 to 90% of the time, OpenScholar achieves citation
  accuracy on par with human experts. OpenScholar's datastore, retriever, and
  self-feedback inference loop also improves off-the-shelf LMs: for instance,
  OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations,
  experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over
  expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's
  32%. We open-source all of our code, models, datastore, data and a public demo.
paper_authors._id: 673ff653b86fd98d952b2afa
paper_authors.name: Akari Asai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:02:01.251Z
paper_authors._id: 6266e0cb7a1f5a1562c4e86e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6266e0cb7a1f5a1562c4e86e/kkJZPssa76mdRPnKH-q3e.jpeg
paper_authors.fullname: Akari Asai
paper_authors.type: user
paper_authors.user: akariasai
paper_authors._id: 673ff653b86fd98d952b2afb
paper_authors.name: Jacqueline He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:01:59.353Z
paper_authors._id: 627c52e51b02d7b5bf671dca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666240522285-627c52e51b02d7b5bf671dca.png
paper_authors.fullname: Jacqueline He
paper_authors.type: user
paper_authors.user: jacquelinehe
paper_authors._id: 673ff653b86fd98d952b2afc
paper_authors.name: Rulin Shao
paper_authors._id: 673ff653b86fd98d952b2afd
paper_authors.name: Weijia Shi
paper_authors._id: 673ff653b86fd98d952b2afe
paper_authors.name: Amanpreet Singh
paper_authors._id: 673ff653b86fd98d952b2aff
paper_authors.name: Joseph Chee Chang
paper_authors._id: 673ff653b86fd98d952b2b00
paper_authors.name: Kyle Lo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:10:05.639Z
paper_authors._id: 670c92d0b6e0a119d335e315
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/hSDQBRzrbGtZawSb-LcGx.png
paper_authors.fullname: Kyle Lo
paper_authors.type: user
paper_authors.user: kylelo
paper_authors._id: 673ff653b86fd98d952b2b01
paper_authors.name: Luca Soldaini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:10:11.852Z
paper_authors._id: 5f04d8c45d08220171a0ad32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
paper_authors.fullname: Luca Soldaini
paper_authors.type: user
paper_authors.user: soldni
paper_authors._id: 673ff653b86fd98d952b2b02
paper_authors.name: Sergey Feldman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:10:17.173Z
paper_authors._id: 6272affe9824228a117849fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6272affe9824228a117849fe/m83mFnu_O8LPem3SmSwRL.jpeg
paper_authors.fullname: Sergey Feldman
paper_authors.type: user
paper_authors.user: sergeyfeldman
paper_authors._id: 673ff653b86fd98d952b2b03
paper_authors.name: Mike D'arcy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:10:22.527Z
paper_authors._id: 659c9bc896d8c1e3dbea8554
paper_authors.avatarUrl: /avatars/0f88ac6e937fcdfad4fc97224d6dd76f.svg
paper_authors.fullname: Mike D'Arcy
paper_authors.type: user
paper_authors.user: miked-ai
paper_authors._id: 673ff653b86fd98d952b2b04
paper_authors.name: David Wadden
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:10:27.652Z
paper_authors._id: 5ff4e2a1463be69ae4bd42bd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1609884433338-5ff4e2a1463be69ae4bd42bd.jpeg
paper_authors.fullname: David Wadden
paper_authors.type: user
paper_authors.user: dwadden
paper_authors._id: 673ff653b86fd98d952b2b05
paper_authors.name: Matt Latzke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:10:33.621Z
paper_authors._id: 65f9dd02d3aad0fbb6dc6b83
paper_authors.avatarUrl: /avatars/8d2d4c5500c13c0631754dd71ae7721f.svg
paper_authors.fullname: Matt Latzke
paper_authors.type: user
paper_authors.user: MLatzke
paper_authors._id: 673ff653b86fd98d952b2b06
paper_authors.name: Minyang Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:10:39.284Z
paper_authors._id: 64ead5f1213a0415bd22d0e4
paper_authors.avatarUrl: /avatars/2797e96d93b999a3e5f816935eb43673.svg
paper_authors.fullname: Minyang Tian
paper_authors.type: user
paper_authors.user: mtian8
paper_authors._id: 673ff653b86fd98d952b2b07
paper_authors.name: Pan Ji
paper_authors._id: 673ff653b86fd98d952b2b08
paper_authors.name: Shengyan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:10:53.076Z
paper_authors._id: 661551375e8f1eaaba4c53b3
paper_authors.avatarUrl: /avatars/1d41150d06ce4e4726269b9c48cbbc10.svg
paper_authors.fullname: LiuShengyang
paper_authors.type: user
paper_authors.user: LSY3579
paper_authors._id: 673ff653b86fd98d952b2b09
paper_authors.name: Hao Tong
paper_authors._id: 673ff653b86fd98d952b2b0a
paper_authors.name: Bohao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:11:03.097Z
paper_authors._id: 6717b1a510c78be6e4335dd4
paper_authors.avatarUrl: /avatars/368e4af45151d2756726f45ec9cb0b64.svg
paper_authors.fullname: Bohao Wu
paper_authors.type: user
paper_authors.user: nlogn-27
paper_authors._id: 673ff653b86fd98d952b2b0b
paper_authors.name: Yanyu Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:11:08.422Z
paper_authors._id: 645862428eef89b5c3c25ba1
paper_authors.avatarUrl: /avatars/b46887a16d17c6e552d26a5a140e81e0.svg
paper_authors.fullname: Yanyu Xiong
paper_authors.type: user
paper_authors.user: yxiong5
paper_authors._id: 673ff653b86fd98d952b2b0c
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 673ff653b86fd98d952b2b0d
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:11:22.628Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_authors._id: 673ff653b86fd98d952b2b0e
paper_authors.name: Dan Weld
paper_authors._id: 673ff653b86fd98d952b2b0f
paper_authors.name: Doug Downey
paper_authors._id: 673ff653b86fd98d952b2b10
paper_authors.name: Wen-tau Yih
paper_authors._id: 673ff653b86fd98d952b2b11
paper_authors.name: Pang Wei Koh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:11:37.747Z
paper_authors._id: 641b4263abfce26bcf7b27de
paper_authors.avatarUrl: /avatars/e91b4205e4f74b0dd8c333c23203a924.svg
paper_authors.fullname: Pang Wei Koh
paper_authors.type: user
paper_authors.user: pangwei
paper_authors._id: 673ff653b86fd98d952b2b12
paper_authors.name: Hannaneh Hajishirzi
paper_title: OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented
    LMs
paper_summary: Scientific progress depends on researchers' ability to synthesize the growing
  body of literature. Can large language models (LMs) assist scientists in this
  task? We introduce OpenScholar, a specialized retrieval-augmented LM that
  answers scientific queries by identifying relevant passages from 45 million
  open-access papers and synthesizing citation-backed responses. To evaluate
  OpenScholar, we develop ScholarQABench, the first large-scale multi-domain
  benchmark for literature search, comprising 2,967 expert-written queries and
  208 long-form answers across computer science, physics, neuroscience, and
  biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and
  PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o
  hallucinates citations 78 to 90% of the time, OpenScholar achieves citation
  accuracy on par with human experts. OpenScholar's datastore, retriever, and
  self-feedback inference loop also improves off-the-shelf LMs: for instance,
  OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations,
  experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over
  expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's
  32%. We open-source all of our code, models, datastore, data and a public demo.
paper_ai_summary: OpenScholar, a specialized language model, enhances scientific query synthesis by retrieving relevant passages and providing citation-backed answers, outperforming existing models in correctness and citation accuracy.
paper_ai_keywords: large language models retrieval-augmented LM passage identification citation-backed responses large-scale benchmark ScholarQABench multi-domain benchmark computer science physics neuroscience biomedicine citation accuracy inference loop off-the-shelf LMs human evaluations