title: SiRA: Sparse Mixture of Low Rank Adaptation
summary: Parameter Efficient Tuning has been an prominent approach to adapt the Large
  Language Model to downstream tasks. Most previous works considers adding the
  dense trainable parameters, where all parameters are used to adapt certain
  task. We found this less effective empirically using the example of LoRA that
  introducing more trainable parameters does not help. Motivated by this we
  investigate the importance of leveraging "sparse" computation and propose SiRA:
  sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of
  Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top
  k experts routing with a capacity limit restricting the maximum number of
  tokens each expert can process. We propose a novel and simple expert dropout on
  top of gating network to reduce the over-fitting issue. Through extensive
  experiments, we verify SiRA performs better than LoRA and other mixture of
  expert approaches across different single tasks and multitask settings.
paper_authors._id: 6555ab2638985840b36fb3c8
paper_authors.name: Yun Zhu
paper_authors._id: 6555ab2638985840b36fb3c9
paper_authors.name: Nevan Wichers
paper_authors._id: 6555ab2638985840b36fb3ca
paper_authors.name: Chu-Cheng Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T10:15:10.635Z
paper_authors._id: 6137b1b20317b0a5c10877c6
paper_authors.avatarUrl: /avatars/8eccd696bffab67e431aed200e96af14.svg
paper_authors.fullname: Chu-Cheng Lin
paper_authors.type: user
paper_authors.user: kitsing
paper_authors._id: 6555ab2638985840b36fb3cb
paper_authors.name: Xinyi Wang
paper_authors._id: 6555ab2638985840b36fb3cc
paper_authors.name: Tianlong Chen
paper_authors._id: 6555ab2638985840b36fb3cd
paper_authors.name: Lei Shu
paper_authors._id: 6555ab2638985840b36fb3ce
paper_authors.name: Han Lu
paper_authors._id: 6555ab2638985840b36fb3cf
paper_authors.name: Canoee Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T10:17:20.424Z
paper_authors._id: 63449f6acbc4ea2ab7405361
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665441596687-noauth.jpeg
paper_authors.fullname: Canoee Liu
paper_authors.type: user
paper_authors.user: Canoee
paper_authors._id: 6555ab2638985840b36fb3d0
paper_authors.name: Liangchen Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T10:17:27.627Z
paper_authors._id: 645a9ab321ab438e732a7b8d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bNp7XtXqPQoauh7CU5B7b.jpeg
paper_authors.fullname: Liangchen Luo
paper_authors.type: user
paper_authors.user: luolc
paper_authors._id: 6555ab2638985840b36fb3d1
paper_authors.name: Jindong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T10:14:46.481Z
paper_authors._id: 635242b8b3678a43742baec6
paper_authors.avatarUrl: /avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
paper_authors.fullname: chenjindong
paper_authors.type: user
paper_authors.user: chenjindong
paper_authors._id: 6555ab2638985840b36fb3d2
paper_authors.name: Lei Meng
paper_title: SiRA: Sparse Mixture of Low Rank Adaptation
paper_summary: Parameter Efficient Tuning has been an prominent approach to adapt the Large
  Language Model to downstream tasks. Most previous works considers adding the
  dense trainable parameters, where all parameters are used to adapt certain
  task. We found this less effective empirically using the example of LoRA that
  introducing more trainable parameters does not help. Motivated by this we
  investigate the importance of leveraging "sparse" computation and propose SiRA:
  sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of
  Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top
  k experts routing with a capacity limit restricting the maximum number of
  tokens each expert can process. We propose a novel and simple expert dropout on
  top of gating network to reduce the over-fitting issue. Through extensive
  experiments, we verify SiRA performs better than LoRA and other mixture of
  expert approaches across different single tasks and multitask settings.
paper_ai_summary: SiRA, a sparse mixture of low-rank adaptation with expert dropout and capacity limits, improves performance over LoRA and other mixture of expert methods in various tasks.
paper_ai_keywords: parameter efficient tuning large language model dense trainable parameters LoRA sparse computation SiRA sparse mixture of low rank adaption Sparse Mixture of Experts SMoE expert dropout gating network multitask settings