Uncovering Limitations of Large Language Models in Information Seeking from Tables
<|reference_start|>Uncovering Limitations of Large Language Models in Information Seeking from Tables: Tables are recognized for their high information density and widespread usage, serving as essential sources of information. Seeking information from tables (TIS) is a crucial capability for Large Language Models (LLMs), serving as the foundation of knowledge-based Q&A systems. However, this field presently suffers from an absence of thorough and reliable evaluation. This paper introduces a more reliable benchmark for Table Information Seeking (TabIS). To avoid the unreliable evaluation caused by text similarity-based metrics, TabIS adopts a single-choice question format (with two options per question) instead of a text generation format. We establish an effective pipeline for generating options, ensuring their difficulty and quality. Experiments conducted on 12 LLMs reveal that while the performance of GPT-4-turbo is marginally satisfactory, both other proprietary and open-source models perform inadequately. Further analysis shows that LLMs exhibit a poor understanding of table structures, and struggle to balance between TIS performance and robustness against pseudo-relevant tables (common in retrieval-augmented systems). These findings uncover the limitations and potential challenges of LLMs in seeking information from tables. We release our data and code to facilitate further research in this field.<|reference_end|>
@article{pang2024uncovering,
    title={Uncovering Limitations of Large Language Models in Information Seeking
  from Tables},
    author={Chaoxu Pang, Yixuan Cao, Chunhao Yang, Ping Luo},
    journal={arXiv preprint arXiv:2406.04113},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.04113},
    primaryClass={cs.CL}
}