MindScope: Exploring cognitive biases in large language models through Multi-Agent Systems
<|reference_start|>MindScope: Exploring cognitive biases in large language models through Multi-Agent Systems: Detecting cognitive biases in large language models (LLMs) is a fascinating task that aims to probe the existing cognitive biases within these models. Current methods for detecting cognitive biases in language models generally suffer from incomplete detection capabilities and a restricted range of detectable bias types. To address this issue, we introduced the 'MindScope' dataset, which distinctively integrates static and dynamic elements. The static component comprises 5,170 open-ended questions spanning 72 cognitive bias categories. The dynamic component leverages a rule-based, multi-agent communication framework to facilitate the generation of multi-round dialogues. This framework is flexible and readily adaptable for various psychological experiments involving LLMs. In addition, we introduce a multi-agent detection method applicable to a wide range of detection tasks, which integrates Retrieval-Augmented Generation (RAG), competitive debate, and a reinforcement learning-based decision module. Demonstrating substantial effectiveness, this method has shown to improve detection accuracy by as much as 35.10% compared to GPT-4. Codes and appendix are available at https://github.com/2279072142/MindScope.<|reference_end|>
@article{xie2024mindscope:,
    title={MindScope: Exploring cognitive biases in large language models through
  Multi-Agent Systems},
    author={Zhentao Xie, Jiabao Zhao, Yilei Wang, Jinxin Shi, Yanhong Bai,
  Xingjiao Wu and Liang He},
    journal={arXiv preprint arXiv:2410.04452},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.04452},
    primaryClass={cs.CL cs.AI}
}