title: Residual Prompt Tuning: Improving Prompt Tuning with Residual
    Reparameterization
summary: Prompt tuning is one of the successful approaches for parameter-efficient
  tuning of pre-trained language models. Despite being arguably the most
  parameter-efficient (tuned soft prompts constitute <0.1% of total parameters),
  it typically performs worse than other efficient tuning methods and is quite
  sensitive to hyper-parameters. In this work, we introduce Residual Prompt
  Tuning - a simple and efficient method that significantly improves the
  performance and stability of prompt tuning. We propose to reparameterize soft
  prompt embeddings using a shallow network with a residual connection. Our
  experiments show that Residual Prompt Tuning significantly outperforms prompt
  tuning on SuperGLUE benchmark. Notably, our method reaches +7 points
  improvement over prompt tuning with T5-Base and allows to reduce the prompt
  length by 10x without hurting performance. In addition, we show that our
  approach is robust to the choice of learning rate and prompt initialization,
  and is effective in few-shot settings.
paper_authors._id: 6459a4663b6fafd9664711e2
paper_authors.name: Anastasia Razdaibiedina
paper_authors._id: 6459a4663b6fafd9664711e3
paper_authors.name: Yuning Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:16:57.213Z
paper_authors._id: 62bf46a630a1888a761f8619
paper_authors.avatarUrl: /avatars/de6c656066b1e88d37882ce68a0fd25c.svg
paper_authors.fullname: Yuning Mao
paper_authors.type: user
paper_authors.user: yuningm
paper_authors._id: 6459a4663b6fafd9664711e4
paper_authors.name: Rui Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:17:13.225Z
paper_authors._id: 6407d236a7bc7c3865adcc3a
paper_authors.avatarUrl: /avatars/8b75cc07fd049acde5363d268c5b88d4.svg
paper_authors.fullname: Rui Hou
paper_authors.type: user
paper_authors.user: rayhou
paper_authors._id: 6459a4663b6fafd9664711e5
paper_authors.name: Madian Khabsa
paper_authors._id: 6459a4663b6fafd9664711e6
paper_authors.name: Mike Lewis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:17:40.021Z
paper_authors._id: 5e67eda71009063689407484
paper_authors.avatarUrl: /avatars/6d7688faba888d463584eaa02ad011e3.svg
paper_authors.fullname: Mike Lewis
paper_authors.type: user
paper_authors.user: mikelewis0
paper_authors._id: 6459a4663b6fafd9664711e7
paper_authors.name: Jimmy Ba
paper_authors._id: 6459a4663b6fafd9664711e8
paper_authors.name: Amjad Almahairi
paper_title: Residual Prompt Tuning: Improving Prompt Tuning with Residual
    Reparameterization
paper_summary: Prompt tuning is one of the successful approaches for parameter-efficient
  tuning of pre-trained language models. Despite being arguably the most
  parameter-efficient (tuned soft prompts constitute <0.1% of total parameters),
  it typically performs worse than other efficient tuning methods and is quite
  sensitive to hyper-parameters. In this work, we introduce Residual Prompt
  Tuning - a simple and efficient method that significantly improves the
  performance and stability of prompt tuning. We propose to reparameterize soft
  prompt embeddings using a shallow network with a residual connection. Our
  experiments show that Residual Prompt Tuning significantly outperforms prompt
  tuning on SuperGLUE benchmark. Notably, our method reaches +7 points
  improvement over prompt tuning with T5-Base and allows to reduce the prompt
  length by 10x without hurting performance. In addition, we show that our
  approach is robust to the choice of learning rate and prompt initialization,
  and is effective in few-shot settings.
paper_ai_summary: Residual Prompt Tuning enhances prompt tuning for parameter-efficient pre-trained language models, improving performance and stability while reducing prompt length.
paper_ai_keywords: prompt tuning parameter-efficient tuning soft prompts reparameterization residual connection SuperGLUE benchmark few-shot settings