Hyper-multi-step: The Truth Behind Difficult Long-context Tasks
<|reference_start|>Hyper-multi-step: The Truth Behind Difficult Long-context Tasks: Long-context language models (LCLM), characterized by their extensive context window, is becoming increasingly popular. Meanwhile, many long-context benchmarks present challenging tasks that even the most advanced LCLMs struggle to complete. However, the underlying sources of various challenging long-context tasks have seldom been studied. To bridge this gap, we conduct experiments to indicate their difficulty stems primarily from two basic issues: "multi-matching retrieval," which requires the simultaneous retrieval of multiple items, and "logic-based retrieval," which necessitates logical judgment within retrieval criteria. These two problems, while seemingly straightforward, actually exceed the capabilities of LCLMs because they are proven to be hyper-multi-step (demanding numerous steps to solve) in nature. This finding could explain why LLMs struggle with more advanced long-context tasks, providing a more accurate perspective for rethinking solutions for them.<|reference_end|>
@article{yu2024hyper-multi-step:,
    title={Hyper-multi-step: The Truth Behind Difficult Long-context Tasks},
    author={Yijiong Yu, Ma Xiufa, Fang Jianwei, Zhi Xu, Su Guangyao, Wang
  Jiancheng, Yongfeng Huang, Zhixiao Qi, Wei Wang, Weifeng Liu, Ran Chen, Ji
  Pei},
    journal={arXiv preprint arXiv:2410.04422},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.04422},
    primaryClass={cs.CL}
}