title: ReviewerGPT? An Exploratory Study on Using Large Language Models for
    Paper Reviewing
summary: Given the rapid ascent of large language models (LLMs), we study the
  question: (How) can large language models help in reviewing of scientific
  papers or proposals? We first conduct some pilot studies where we find that (i)
  GPT-4 outperforms other LLMs (Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly,
  OpenAssistant, StableLM), and (ii) prompting with a specific question (e.g., to
  identify errors) outperforms prompting to simply write a review. With these
  insights, we study the use of LLMs (specifically, GPT-4) for three tasks:
    1. Identifying errors: We construct 13 short computer science papers each
  with a deliberately inserted error, and ask the LLM to check for the
  correctness of these papers. We observe that the LLM finds errors in 7 of them,
  spanning both mathematical and conceptual errors.
    2. Verifying checklists: We task the LLM to verify 16 closed-ended checklist
  questions in the respective sections of 15 NeurIPS 2022 papers. We find that
  across 119 {checklist question, paper} pairs, the LLM had an 86.6% accuracy.
    3. Choosing the "better" paper: We generate 10 pairs of abstracts,
  deliberately designing each pair in such a way that one abstract was clearly
  superior than the other. The LLM, however, struggled to discern these
  relatively straightforward distinctions accurately, committing errors in its
  evaluations for 6 out of the 10 pairs.
    Based on these experiments, we think that LLMs have a promising use as
  reviewing assistants for specific reviewing tasks, but not (yet) for complete
  evaluations of papers or proposals.
paper_authors._id: 64793f55ac77bb7603846fc5
paper_authors.name: Ryan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:06:36.316Z
paper_authors._id: 647a12028d17fcf64761a3fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3GEiY7YCU_FzJing5wP-C.jpeg
paper_authors.fullname: Ryan Liu
paper_authors.type: user
paper_authors.user: theryanliu
paper_authors._id: 64793f55ac77bb7603846fc6
paper_authors.name: Nihar B. Shah
paper_title: ReviewerGPT? An Exploratory Study on Using Large Language Models for
    Paper Reviewing
paper_summary: Given the rapid ascent of large language models (LLMs), we study the
  question: (How) can large language models help in reviewing of scientific
  papers or proposals? We first conduct some pilot studies where we find that (i)
  GPT-4 outperforms other LLMs (Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly,
  OpenAssistant, StableLM), and (ii) prompting with a specific question (e.g., to
  identify errors) outperforms prompting to simply write a review. With these
  insights, we study the use of LLMs (specifically, GPT-4) for three tasks:
    1. Identifying errors: We construct 13 short computer science papers each
  with a deliberately inserted error, and ask the LLM to check for the
  correctness of these papers. We observe that the LLM finds errors in 7 of them,
  spanning both mathematical and conceptual errors.
    2. Verifying checklists: We task the LLM to verify 16 closed-ended checklist
  questions in the respective sections of 15 NeurIPS 2022 papers. We find that
  across 119 {checklist question, paper} pairs, the LLM had an 86.6% accuracy.
    3. Choosing the "better" paper: We generate 10 pairs of abstracts,
  deliberately designing each pair in such a way that one abstract was clearly
  superior than the other. The LLM, however, struggled to discern these
  relatively straightforward distinctions accurately, committing errors in its
  evaluations for 6 out of the 10 pairs.
    Based on these experiments, we think that LLMs have a promising use as
  reviewing assistants for specific reviewing tasks, but not (yet) for complete
  evaluations of papers or proposals.
paper_ai_summary: Large language models, particularly GPT-4, demonstrated effectiveness in identifying errors and verifying checklist questions in scientific papers, but struggled with comparing and choosing the better paper.
paper_ai_keywords: GPT-4 large language models LLMs Bard Vicuna Koala Alpaca LLaMa Dolly OpenAssistant StableLM error identification checklist verification NeurIPS 2022