ResearchArena: Benchmarking LLMs' Ability to Collect and Organize Information as Research Agents
<|reference_start|>ResearchArena: Benchmarking LLMs' Ability to Collect and Organize Information as Research Agents: Large language models (LLMs) have exhibited remarkable performance across various tasks in natural language processing. Nevertheless, challenges still arise when these tasks demand domain-specific expertise and advanced analytical skills, such as conducting research surveys on a designated topic. In this research, we develop ResearchArena, a benchmark that measures LLM agents' ability to conduct academic surveys, an initial step of academic research process. Specifically, we deconstructs the surveying process into three stages 1) information discovery: locating relevant papers, 2) information selection: assessing papers' importance to the topic, and 3) information organization: organizing papers into meaningful structures. In particular, we establish an offline environment comprising 12.0M full-text academic papers and 7.9K survey papers, which evaluates agents' ability to locate supporting materials for composing the survey on a topic, rank the located papers based on their impact, and organize these into a hierarchical knowledge mind-map. With this benchmark, we conduct preliminary evaluations of existing techniques and find that all LLM-based methods under-performing when compared to basic keyword-based retrieval techniques, highlighting substantial opportunities for future research.<|reference_end|>
@article{kang2024researcharena:,
    title={ResearchArena: Benchmarking LLMs' Ability to Collect and Organize
  Information as Research Agents},
    author={Hao Kang, Chenyan Xiong},
    journal={arXiv preprint arXiv:2406.10291},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.10291},
    primaryClass={cs.AI cs.CL cs.IR}
}