Making Long-Context Language Models Better Multi-Hop Reasoners
<|reference_start|>Making Long-Context Language Models Better Multi-Hop Reasoners: Recent advancements in long-context modeling have enhanced language models (LMs) for complex tasks across multiple NLP applications. Despite this progress, we find that these models struggle with multi-hop reasoning and exhibit decreased performance in the presence of noisy contexts. In this paper, we introduce Reasoning with Attributions, a novel approach that prompts LMs to supply attributions for each assertion during their reasoning. We validate our approach through experiments on three multi-hop datasets, employing both proprietary and open-source models, and demonstrate its efficacy and resilience. Furthermore, we explore methods to augment reasoning capabilities via fine-tuning and offer an attribution-annotated dataset and a specialized training strategy. Our fine-tuned model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant.<|reference_end|>
@article{li2024making,
    title={Making Long-Context Language Models Better Multi-Hop Reasoners},
    author={Yanyang Li, Shuo Liang, Michael R. Lyu, Liwei Wang},
    journal={arXiv preprint arXiv:2408.03246},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.03246},
    primaryClass={cs.CL}
}