title: Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
summary: Large foundation models are becoming ubiquitous, but training them from
  scratch is prohibitively expensive. Thus, efficiently adapting these powerful
  models to downstream tasks is increasingly important. In this paper, we study a
  principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream
  task adaptation. Despite demonstrating good generalizability, OFT still uses a
  fairly large number of trainable parameters due to the high dimensionality of
  orthogonal matrices. To address this, we start by examining OFT from an
  information transmission perspective, and then identify a few key desiderata
  that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast
  Fourier transform algorithm enables efficient information transmission, we
  propose an efficient orthogonal parameterization using butterfly structures. We
  apply this parameterization to OFT, creating a novel parameter-efficient
  finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a
  special case, BOFT introduces a generalized orthogonal finetuning framework.
  Finally, we conduct an extensive empirical study of adapting large vision
  transformers, large language models, and text-to-image diffusion models to
  various downstream tasks in vision and language.
paper_authors._id: 65519c62a43b13ee347a088a
paper_authors.name: Weiyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T08:47:18.767Z
paper_authors._id: 648905d1a15c43c791d4381f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
paper_authors.fullname: Weiyang Liu
paper_authors.type: user
paper_authors.user: wy1iu
paper_authors._id: 65519c62a43b13ee347a088b
paper_authors.name: Zeju Qiu
paper_authors._id: 65519c62a43b13ee347a088c
paper_authors.name: Yao Feng
paper_authors._id: 65519c62a43b13ee347a088d
paper_authors.name: Yuliang Xiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T08:48:39.195Z
paper_authors._id: 6183d0b249ef1d984699e4a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6183d0b249ef1d984699e4a3/GYR4kzPUEJPxrnQD-trhT.png
paper_authors.fullname: Yuliang Xiu
paper_authors.type: user
paper_authors.user: Yuliang
paper_authors._id: 65519c62a43b13ee347a088e
paper_authors.name: Yuxuan Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T08:49:28.761Z
paper_authors._id: 641f12813c49fc33ced76018
paper_authors.avatarUrl: /avatars/46f23d38364f86bdd445836f4ad155ec.svg
paper_authors.fullname: yuxuanxue
paper_authors.type: user
paper_authors.user: yuxuanxue
paper_authors._id: 65519c62a43b13ee347a088f
paper_authors.name: Longhui Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T08:49:41.033Z
paper_authors._id: 64b753306c169983c982f609
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b753306c169983c982f609/oOC_N8cI1Go6avUMvIPVI.jpeg
paper_authors.fullname: Longhui Yu
paper_authors.type: user
paper_authors.user: Longhui98
paper_authors._id: 65519c62a43b13ee347a0890
paper_authors.name: Haiwen Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T13:24:43.811Z
paper_authors._id: 644d2ede5c1b4e14d0c065af
paper_authors.avatarUrl: /avatars/34945e0ab925d8eabc6580c71e053b11.svg
paper_authors.fullname: Haiwen (Haven) Feng
paper_authors.type: user
paper_authors.user: havenfeng
paper_authors._id: 65519c62a43b13ee347a0891
paper_authors.name: Zhen Liu
paper_authors._id: 65519c62a43b13ee347a0892
paper_authors.name: Juyeon Heo
paper_authors._id: 65519c62a43b13ee347a0893
paper_authors.name: Songyou Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T08:52:42.204Z
paper_authors._id: 620cb0722d8bc91e1cf2e6e7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644998719617-noauth.jpeg
paper_authors.fullname: Songyou Peng
paper_authors.type: user
paper_authors.user: syp
paper_authors._id: 65519c62a43b13ee347a0894
paper_authors.name: Yandong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T08:52:51.533Z
paper_authors._id: 643c21735fcffe09fb68a46f
paper_authors.avatarUrl: /avatars/76aabacd318aa954d4c53094ad456056.svg
paper_authors.fullname: Yandong Wen
paper_authors.type: user
paper_authors.user: ydwen
paper_authors._id: 65519c62a43b13ee347a0895
paper_authors.name: Michael J. Black
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T08:52:59.779Z
paper_authors._id: 649fbababa7a05e3626546c8
paper_authors.avatarUrl: /avatars/a9aa4b4914649527094fc1b12779e014.svg
paper_authors.fullname: Michael J. Black
paper_authors.type: user
paper_authors.user: MichaelJBlack
paper_authors._id: 65519c62a43b13ee347a0896
paper_authors.name: Adrian Weller
paper_authors._id: 65519c62a43b13ee347a0897
paper_authors.name: Bernhard Schölkopf
paper_title: Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
paper_summary: Large foundation models are becoming ubiquitous, but training them from
  scratch is prohibitively expensive. Thus, efficiently adapting these powerful
  models to downstream tasks is increasingly important. In this paper, we study a
  principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream
  task adaptation. Despite demonstrating good generalizability, OFT still uses a
  fairly large number of trainable parameters due to the high dimensionality of
  orthogonal matrices. To address this, we start by examining OFT from an
  information transmission perspective, and then identify a few key desiderata
  that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast
  Fourier transform algorithm enables efficient information transmission, we
  propose an efficient orthogonal parameterization using butterfly structures. We
  apply this parameterization to OFT, creating a novel parameter-efficient
  finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a
  special case, BOFT introduces a generalized orthogonal finetuning framework.
  Finally, we conduct an extensive empirical study of adapting large vision
  transformers, large language models, and text-to-image diffusion models to
  various downstream tasks in vision and language.
paper_ai_keywords: Orthogonal Finetuning (OFT) orthogonal matrices information transmission butterfly structures Orthogonal Butterfly (BOFT) generalized orthogonal finetuning framework vision transformers language models text-to-image diffusion models