title: State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for
    State Space Models
summary: State Space Models (SSMs) have emerged as efficient alternatives to
  Transformers, mitigating their quadratic computational cost. However, the
  application of Parameter-Efficient Fine-Tuning (PEFT) methods to SSMs remains
  largely unexplored. In particular, prompt-based methods like Prompt Tuning and
  Prefix-Tuning, which are widely used in Transformers, do not perform well on
  SSMs. To address this, we propose state-based methods as a superior alternative
  to prompt-based methods. This new family of methods naturally stems from the
  architectural characteristics of SSMs. State-based methods adjust state-related
  features directly instead of depending on external prompts. Furthermore, we
  introduce a novel state-based PEFT method: State-offset Tuning. At every
  timestep, our method directly affects the state at the current step, leading to
  more effective adaptation. Through extensive experiments across diverse
  datasets, we demonstrate the effectiveness of our method. Code is available at
  https://github.com/furiosa-ai/ssm-state-tuning.
paper_authors._id: 67cb02680a2a716f25805cb4
paper_authors.name: Wonjun Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:54:53.750Z
paper_authors._id: 63e1d5247fbb6ae4d4f4cc8e
paper_authors.avatarUrl: /avatars/8a8f700adf9e8000641c2c2f6bd56080.svg
paper_authors.fullname: Wonjun Kang
paper_authors.type: user
paper_authors.user: wjkang
paper_authors._id: 67cb02680a2a716f25805cb5
paper_authors.name: Kevin Galim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:55:00.348Z
paper_authors._id: 630c90123dc31beba6e8f406
paper_authors.avatarUrl: /avatars/2188b41fff122d4f5683b46c529ed79d.svg
paper_authors.fullname: Kevin Galim
paper_authors.type: user
paper_authors.user: kev95
paper_authors._id: 67cb02680a2a716f25805cb6
paper_authors.name: Yuchen Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:55:07.129Z
paper_authors._id: 65ded6ec6efc2a4ebb18d306
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/SPsUBPd_CKMBzACSCgwP5.png
paper_authors.fullname: Yuchen Zeng
paper_authors.type: user
paper_authors.user: yzeng58
paper_authors._id: 67cb02680a2a716f25805cb7
paper_authors.name: Minjae Lee
paper_authors._id: 67cb02680a2a716f25805cb8
paper_authors.name: Hyung Il Koo
paper_authors._id: 67cb02680a2a716f25805cb9
paper_authors.name: Nam Ik Cho
paper_title: State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for
    State Space Models
paper_summary: State Space Models (SSMs) have emerged as efficient alternatives to
  Transformers, mitigating their quadratic computational cost. However, the
  application of Parameter-Efficient Fine-Tuning (PEFT) methods to SSMs remains
  largely unexplored. In particular, prompt-based methods like Prompt Tuning and
  Prefix-Tuning, which are widely used in Transformers, do not perform well on
  SSMs. To address this, we propose state-based methods as a superior alternative
  to prompt-based methods. This new family of methods naturally stems from the
  architectural characteristics of SSMs. State-based methods adjust state-related
  features directly instead of depending on external prompts. Furthermore, we
  introduce a novel state-based PEFT method: State-offset Tuning. At every
  timestep, our method directly affects the state at the current step, leading to
  more effective adaptation. Through extensive experiments across diverse
  datasets, we demonstrate the effectiveness of our method. Code is available at
  https://github.com/furiosa-ai/ssm-state-tuning.
paper_ai_keywords: State Space Models (SSMs) Parameter-Efficient Fine-Tuning (PEFT) Prompt Tuning Prefix-Tuning State-based methods State-offset Tuning timesteps state-related features state-at-the-current-step