Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions
<|reference_start|>Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions: Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields and have shown significant potential in the academic peer-review process. However, existing applications are primarily limited to static review generation based on submitted papers, which fail to capture the dynamic and iterative nature of real-world peer reviews. In this paper, we reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers. We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources, including the top-tier conference and prestigious journal. This dataset is meticulously designed to facilitate the applications of LLMs for multi-turn dialogues, effectively simulating the complete peer-review process. Furthermore, we propose a series of metrics to evaluate the performance of LLMs for each role under this reformulated peer-review setting, ensuring fair and comprehensive evaluations. We believe this work provides a promising perspective on enhancing the LLM-driven peer-review process by incorporating dynamic, role-based interactions. It aligns closely with the iterative and interactive nature of real-world academic peer review, offering a robust foundation for future research and development in this area. We open-source the dataset at https://github.com/chengtan9907/ReviewMT.<|reference_end|>
@article{tan2024peer,
    title={Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based
  Interactions},
    author={Cheng Tan, Dongxin Lyu, Siyuan Li, Zhangyang Gao, Jingxuan Wei, Siqi
  Ma, Zicheng Liu, Stan Z. Li},
    journal={arXiv preprint arXiv:2406.05688},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.05688},
    primaryClass={cs.CL cs.AI cs.LG}
}