FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition
<|reference_start|>FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition: Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. However, such a paradigm fails to comprehensively differentiate the fine-grained language and cognitive skills, rendering the lack of sufficient interpretation to LLMs' capabilities. In this paper, we present FAC$^2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation. Specifically, we formulate LLMs' evaluation in a multi-dimensional and explainable manner by dissociating the language-related capabilities and the cognition-related ones. Besides, through extracting the intermediate reasoning from LLMs, we further break down the process of applying a specific capability into three sub-steps: recalling relevant knowledge, utilizing knowledge, and solving problems. Finally, FAC$^2$E evaluates each sub-step of each fine-grained capability, providing a two-faceted diagnosis for LLMs. Utilizing FAC$^2$E, we identify a common shortfall in knowledge utilization among models and propose a straightforward, knowledge-enhanced method to mitigate this issue. Our results not only showcase promising performance enhancements but also highlight a direction for future LLM advancements.<|reference_end|>
@article{wang2024fac$^2$e:,
    title={FAC$^2$E: Better Understanding Large Language Model Capabilities by
  Dissociating Language and Cognition},
    author={Xiaoqiang Wang, Lingfei Wu, Tengfei Ma and Bang Liu},
    journal={arXiv preprint arXiv:2403.00126},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.00126},
    primaryClass={cs.CL}
}