title: LongAgent: Scaling Language Models to 128k Context through Multi-Agent
    Collaboration
summary: Large language models (LLMs) have demonstrated impressive performance in
  understanding language and executing complex reasoning tasks. However, LLMs
  with long context windows have been notorious for their expensive training
  costs and high inference latency. Even the most advanced models such as GPT-4
  and Claude2 often make mistakes when processing inputs of over 100k tokens, a
  phenomenon also known as lost in the middle. In this paper, we propose
  LongAgent, a method based on multi-agent collaboration, which scales
  LLMs (e.g., LLaMA) to a context of 128K and demonstrates potential superiority
  in long-text processing compared to GPT-4. In LongAgent, a leader is
  responsible for understanding user intent and directing team members to acquire
  information from documents. Due to members' hallucinations, it is non-trivial
  for a leader to obtain accurate information from the responses of dozens to
  hundreds of members. To address this, we develop an inter-member
  communication mechanism to resolve response conflicts caused by hallucinations
  through information sharing. Our experimental results indicate that
  LongAgent offers a promising alternative for long-text processing. The
  agent team instantiated with LLaMA-7B achieves significant improvements in
  tasks such as 128k-long text retrieval, multi-hop question answering, compared
  to GPT-4.
paper_authors._id: 65d42906cf4034c90c00f943
paper_authors.name: Jun Zhao
paper_authors._id: 65d42906cf4034c90c00f944
paper_authors.name: Can Zu
paper_authors._id: 65d42906cf4034c90c00f945
paper_authors.name: Hao Xu
paper_authors._id: 65d42906cf4034c90c00f946
paper_authors.name: Yi Lu
paper_authors._id: 65d42906cf4034c90c00f947
paper_authors.name: Wei He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:32:16.410Z
paper_authors._id: 66ecee857264238429a1211f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66ecee857264238429a1211f/TbuM7ToLBrSxDF8mOccpK.jpeg
paper_authors.fullname: Wei He
paper_authors.type: user
paper_authors.user: hewei2001
paper_authors._id: 65d42906cf4034c90c00f948
paper_authors.name: Yiwen Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T06:59:56.763Z
paper_authors._id: 650819ad95c45596854271a3
paper_authors.avatarUrl: /avatars/9517fd62eef104c33884dfb3f64249bf.svg
paper_authors.fullname: wen
paper_authors.type: user
paper_authors.user: cindywen
paper_authors._id: 65d42906cf4034c90c00f949
paper_authors.name: Tao Gui
paper_authors._id: 65d42906cf4034c90c00f94a
paper_authors.name: Qi Zhang
paper_authors._id: 65d42906cf4034c90c00f94b
paper_authors.name: Xuanjing Huang
paper_title: LongAgent: Scaling Language Models to 128k Context through Multi-Agent
    Collaboration
paper_summary: Large language models (LLMs) have demonstrated impressive performance in
  understanding language and executing complex reasoning tasks. However, LLMs
  with long context windows have been notorious for their expensive training
  costs and high inference latency. Even the most advanced models such as GPT-4
  and Claude2 often make mistakes when processing inputs of over 100k tokens, a
  phenomenon also known as lost in the middle. In this paper, we propose
  LongAgent, a method based on multi-agent collaboration, which scales
  LLMs (e.g., LLaMA) to a context of 128K and demonstrates potential superiority
  in long-text processing compared to GPT-4. In LongAgent, a leader is
  responsible for understanding user intent and directing team members to acquire
  information from documents. Due to members' hallucinations, it is non-trivial
  for a leader to obtain accurate information from the responses of dozens to
  hundreds of members. To address this, we develop an inter-member
  communication mechanism to resolve response conflicts caused by hallucinations
  through information sharing. Our experimental results indicate that
  LongAgent offers a promising alternative for long-text processing. The
  agent team instantiated with LLaMA-7B achieves significant improvements in
  tasks such as 128k-long text retrieval, multi-hop question answering, compared
  to GPT-4.
paper_ai_summary: LongAgent, a multi-agent collaboration method, scales large language models to handle up to 128K tokens with improved accuracy and performance in long-text processing tasks compared to GPT-4.
paper_ai_keywords: multi-agent collaboration context window hallucinations inter-member communication long-text processing LLaMA text retrieval multi-hop question answering