title: Large Language Models Can Self-Improve in Long-context Reasoning
summary: Large language models (LLMs) have achieved substantial progress in processing
  long contexts but still struggle with long-context reasoning. Existing
  approaches typically involve fine-tuning LLMs with synthetic data, which
  depends on annotations from human experts or advanced models like GPT-4, thus
  restricting further advancements. To address this issue, we investigate the
  potential for LLMs to self-improve in long-context reasoning and propose \ours,
  an approach specifically designed for this purpose. This approach is
  straightforward: we sample multiple outputs for each question, score them with
  Minimum Bayes Risk, and then apply supervised fine-tuning or preference
  optimization based on these outputs. Extensive experiments on several leading
  LLMs demonstrate the effectiveness of \ours, with an absolute improvement of
  4.2 points for Llama-3.1-8B-Instruct. Furthermore, \ours achieves superior
  performance compared to prior approaches that depend on data produced by human
  experts or advanced models. We anticipate that this work will open new avenues
  for self-improvement techniques in long-context scenarios, which are essential
  for the continual advancement of LLMs.
paper_authors._id: 673593ee8d2bfc6159c2f994
paper_authors.name: Siheng Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-14T06:09:11.650Z
paper_authors._id: 66d45a8de5837f38ce3b73f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66d45a8de5837f38ce3b73f7/5MQusIK5fD6W9kJwdXIlR.jpeg
paper_authors.fullname: SihengLi
paper_authors.type: user
paper_authors.user: Siheng99
paper_authors._id: 673593ee8d2bfc6159c2f995
paper_authors.name: Cheng Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-17T16:18:14.444Z
paper_authors._id: 64b75db417570fdff9b43fd9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b75db417570fdff9b43fd9/-NcNL_QzuiAAvsBIQG3xo.png
paper_authors.fullname: Cheng YANG
paper_authors.type: user
paper_authors.user: Cheng-YANG
paper_authors._id: 673593ee8d2bfc6159c2f996
paper_authors.name: Zesen Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:21:54.860Z
paper_authors._id: 65b2529285b6c21448a10d65
paper_authors.avatarUrl: /avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
paper_authors.fullname: Zesen Cheng
paper_authors.type: user
paper_authors.user: ClownRat
paper_authors._id: 673593ee8d2bfc6159c2f997
paper_authors.name: Lemao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:09:49.517Z
paper_authors._id: 64462d77efa0723f3946356b
paper_authors.avatarUrl: /avatars/f9757030d82c69aef933309e0c83ccd0.svg
paper_authors.fullname: Lemao Liu
paper_authors.type: user
paper_authors.user: lemaoliu
paper_authors._id: 673593ee8d2bfc6159c2f998
paper_authors.name: Mo Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:36:25.917Z
paper_authors._id: 67af92045a86287292026808
paper_authors.avatarUrl: /avatars/8bad9272fe73ba04e077b5484837c8d3.svg
paper_authors.fullname: Mo
paper_authors.type: user
paper_authors.user: BishopGorov
paper_authors._id: 673593ee8d2bfc6159c2f999
paper_authors.name: Yujiu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:10:07.290Z
paper_authors._id: 64ca1fe838837b12d5e529b7
paper_authors.avatarUrl: /avatars/44a3ad9e59318784ac531993b5f69f6b.svg
paper_authors.fullname: Yujiu Yang
paper_authors.type: user
paper_authors.user: Thu-redrobot
paper_authors._id: 673593ee8d2bfc6159c2f99a
paper_authors.name: Wai Lam
paper_title: Large Language Models Can Self-Improve in Long-context Reasoning
paper_summary: Large language models (LLMs) have achieved substantial progress in processing
  long contexts but still struggle with long-context reasoning. Existing
  approaches typically involve fine-tuning LLMs with synthetic data, which
  depends on annotations from human experts or advanced models like GPT-4, thus
  restricting further advancements. To address this issue, we investigate the
  potential for LLMs to self-improve in long-context reasoning and propose \ours,
  an approach specifically designed for this purpose. This approach is
  straightforward: we sample multiple outputs for each question, score them with
  Minimum Bayes Risk, and then apply supervised fine-tuning or preference
  optimization based on these outputs. Extensive experiments on several leading
  LLMs demonstrate the effectiveness of \ours, with an absolute improvement of
  4.2 points for Llama-3.1-8B-Instruct. Furthermore, \ours achieves superior
  performance compared to prior approaches that depend on data produced by human
  experts or advanced models. We anticipate that this work will open new avenues
  for self-improvement techniques in long-context scenarios, which are essential
  for the continual advancement of LLMs.
paper_ai_keywords: Minimum Bayes Risk supervised fine-tuning preference optimization