Evaluating LLMs with Multiple Problems at once: A New Paradigm for Probing LLM Capabilities
<|reference_start|>Evaluating LLMs with Multiple Problems at once: A New Paradigm for Probing LLM Capabilities: Current LLM evaluation predominantly performs evaluation with prompts comprising single problems. We propose multi-problem evaluation as an additional approach to study the multiple problem handling capabilities of LLMs. We present a systematic study in this regard by comprehensively examining 7 LLMs on 4 related types of tasks constructed from 6 classification benchmarks. The 4 task types include traditional single-problem tasks, homogeneous multi-problem tasks, and two index selection tasks that embed the multi-problem tasks. We find that LLMs are competent multi-problem solvers: they generally perform (nearly) as well on multi-problem tasks as on single-problem tasks. Furthermore, contrary to common expectation, they often do not suffer from a positional bias with long inputs. This makes multi-problem prompting a simple and cost-efficient prompting method of practical significance. However, our results also strongly indicate that LLMs lack true understanding: they perform significantly worse in the two index selection tasks than in the multi-problem task under various evaluation settings, although they can indeed do index selection in general.<|reference_end|>
@article{wang2024exploring,
    title={Exploring the Zero-Shot Capabilities of LLMs Handling Multiple Problems
  at once},
    author={Zhengxiang Wang, Jordan Kodner, Owen Rambow},
    journal={arXiv preprint arXiv:2406.10786},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.10786},
    primaryClass={cs.AI cs.CL}
}