title: Understanding and Mitigating Bottlenecks of State Space Models through
    the Lens of Recency and Over-smoothing
summary: Structured State Space Models (SSMs) have emerged as alternatives to
  transformers. While SSMs are often regarded as effective in capturing
  long-sequence dependencies, we rigorously demonstrate that they are inherently
  limited by strong recency bias. Our empirical studies also reveal that this
  bias impairs the models' ability to recall distant information and introduces
  robustness issues. Our scaling experiments then discovered that deeper
  structures in SSMs can facilitate the learning of long contexts. However,
  subsequent theoretical analysis reveals that as SSMs increase in depth, they
  exhibit another inevitable tendency toward over-smoothing, e.g., token
  representations becoming increasingly indistinguishable. This fundamental
  dilemma between recency and over-smoothing hinders the scalability of existing
  SSMs. Inspired by our theoretical findings, we propose to polarize two channels
  of the state transition matrices in SSMs, setting them to zero and one,
  respectively, simultaneously addressing recency bias and over-smoothing.
  Experiments demonstrate that our polarization technique consistently enhances
  the associative recall accuracy of long-range tokens and unlocks SSMs to
  benefit further from deeper architectures. All source codes are released at
  https://github.com/VITA-Group/SSM-Bottleneck.
paper_authors._id: 67777d99f215dad5eb2e85c6
paper_authors.name: Peihao Wang
paper_authors._id: 67777d99f215dad5eb2e85c7
paper_authors.name: Ruisi Cai
paper_authors._id: 67777d99f215dad5eb2e85c8
paper_authors.name: Yuehao Wang
paper_authors._id: 67777d99f215dad5eb2e85c9
paper_authors.name: Jiajun Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:05.945Z
paper_authors._id: 65dc42ad1ed8dc6d82e3e4d8
paper_authors.avatarUrl: /avatars/d2001d58b4fff24509c9e8e952042fe5.svg
paper_authors.fullname: Jiajun Zhu
paper_authors.type: user
paper_authors.user: lanczos
paper_authors._id: 67777d99f215dad5eb2e85ca
paper_authors.name: Pragya Srivastava
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-03T06:05:01.330Z
paper_authors._id: 639ccab166106be1436e1640
paper_authors.avatarUrl: /avatars/1e3806e18ac427be20e93e5400f153d4.svg
paper_authors.fullname: Pragya Srivastava
paper_authors.type: user
paper_authors.user: pragsri8
paper_authors._id: 67777d99f215dad5eb2e85cb
paper_authors.name: Zhangyang Wang
paper_authors._id: 67777d99f215dad5eb2e85cc
paper_authors.name: Pan Li
paper_title: Understanding and Mitigating Bottlenecks of State Space Models through
    the Lens of Recency and Over-smoothing
paper_summary: Structured State Space Models (SSMs) have emerged as alternatives to
  transformers. While SSMs are often regarded as effective in capturing
  long-sequence dependencies, we rigorously demonstrate that they are inherently
  limited by strong recency bias. Our empirical studies also reveal that this
  bias impairs the models' ability to recall distant information and introduces
  robustness issues. Our scaling experiments then discovered that deeper
  structures in SSMs can facilitate the learning of long contexts. However,
  subsequent theoretical analysis reveals that as SSMs increase in depth, they
  exhibit another inevitable tendency toward over-smoothing, e.g., token
  representations becoming increasingly indistinguishable. This fundamental
  dilemma between recency and over-smoothing hinders the scalability of existing
  SSMs. Inspired by our theoretical findings, we propose to polarize two channels
  of the state transition matrices in SSMs, setting them to zero and one,
  respectively, simultaneously addressing recency bias and over-smoothing.
  Experiments demonstrate that our polarization technique consistently enhances
  the associative recall accuracy of long-range tokens and unlocks SSMs to
  benefit further from deeper architectures. All source codes are released at
  https://github.com/VITA-Group/SSM-Bottleneck.
paper_ai_keywords: Rapid State Space Models (SSMs) long-sequence dependencies recency bias distant information robustness issues over-smoothing token representations state transition matrices polarization technique associative recall accuracy long-range tokens