title: DenseMamba: State Space Models with Dense Hidden Connection for
    Efficient Large Language Models
summary: Large language models (LLMs) face a daunting challenge due to the excessive
  computational and memory requirements of the commonly used Transformer
  architecture. While state space model (SSM) is a new type of foundational
  network architecture offering lower computational complexity, their performance
  has yet to fully rival that of Transformers. This paper introduces DenseSSM, a
  novel approach to enhance the flow of hidden information between layers in
  SSMs. By selectively integrating shallowlayer hidden states into deeper layers,
  DenseSSM retains fine-grained information crucial for the final output. Dense
  connections enhanced DenseSSM still maintains the training parallelizability
  and inference efficiency. The proposed method can be widely applicable to
  various SSM types like RetNet and Mamba. With similar model size, DenseSSM
  achieves significant improvements, exemplified by DenseRetNet outperforming the
  original RetNet with up to 5% accuracy improvement on public benchmarks.
paper_authors._id: 65e782203b2aca8cd1e3c8ae
paper_authors.name: Wei He
paper_authors._id: 65e782203b2aca8cd1e3c8af
paper_authors.name: Kai Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:15:25.683Z
paper_authors._id: 65e52e7d27dc8aa470a640e3
paper_authors.avatarUrl: /avatars/022a179d14de29b9ab9d96fcc85aa264.svg
paper_authors.fullname: hankai
paper_authors.type: user
paper_authors.user: hankaixyz
paper_authors._id: 65e782203b2aca8cd1e3c8b0
paper_authors.name: Yehui Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-05T21:00:10.881Z
paper_authors._id: 64d5deb154bb9eb704f83122
paper_authors.avatarUrl: /avatars/86ce09bcca903319051e2307581a43f4.svg
paper_authors.fullname: Yehui Tang
paper_authors.type: user
paper_authors.user: tangyehui
paper_authors._id: 65e782203b2aca8cd1e3c8b1
paper_authors.name: Chengcheng Wang
paper_authors._id: 65e782203b2aca8cd1e3c8b2
paper_authors.name: Yujie Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-05T21:00:57.559Z
paper_authors._id: 654d989f636d5f1b88a734f1
paper_authors.avatarUrl: /avatars/f72e6c44b1c47b4a125370a7bf6e52c7.svg
paper_authors.fullname: Yujie Yang
paper_authors.type: user
paper_authors.user: EvanYYJ
paper_authors._id: 65e782203b2aca8cd1e3c8b3
paper_authors.name: Tianyu Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-05T21:01:04.778Z
paper_authors._id: 62dd692c26b500df9124b2eb
paper_authors.avatarUrl: /avatars/a490194a2839904e9497388bacf00c10.svg
paper_authors.fullname: Tianyu Guo
paper_authors.type: user
paper_authors.user: tiguo66
paper_authors._id: 65e782203b2aca8cd1e3c8b4
paper_authors.name: Yunhe Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-05T21:01:10.813Z
paper_authors._id: 658bdf7b925aadd43304f05c
paper_authors.avatarUrl: /avatars/64d9e9dea27c376c3bc7b2a54efc2a46.svg
paper_authors.fullname: Yunhe Wang
paper_authors.type: user
paper_authors.user: MightyCrane
paper_title: DenseMamba: State Space Models with Dense Hidden Connection for
    Efficient Large Language Models
paper_summary: Large language models (LLMs) face a daunting challenge due to the excessive
  computational and memory requirements of the commonly used Transformer
  architecture. While state space model (SSM) is a new type of foundational
  network architecture offering lower computational complexity, their performance
  has yet to fully rival that of Transformers. This paper introduces DenseSSM, a
  novel approach to enhance the flow of hidden information between layers in
  SSMs. By selectively integrating shallowlayer hidden states into deeper layers,
  DenseSSM retains fine-grained information crucial for the final output. Dense
  connections enhanced DenseSSM still maintains the training parallelizability
  and inference efficiency. The proposed method can be widely applicable to
  various SSM types like RetNet and Mamba. With similar model size, DenseSSM
  achieves significant improvements, exemplified by DenseRetNet outperforming the
  original RetNet with up to 5% accuracy improvement on public benchmarks.
paper_ai_keywords: large language models (LLMs) Transformer architecture state space model (SSM) DenseSSM hidden information shallowlayer hidden states fine-grained information training parallelizability inference efficiency RetNet Mamba DenseRetNet accuracy improvement public benchmarks