title: SHERL: Synthesizing High Accuracy and Efficient Memory for
    Resource-Limited Transfer Learning
summary: Parameter-efficient transfer learning (PETL) has emerged as a flourishing
  research field for adapting large pre-trained models to downstream tasks,
  greatly reducing trainable parameters while grappling with memory challenges
  during fine-tuning. To address it, memory-efficient series (METL) avoid
  backpropagating gradients through the large backbone. However, they compromise
  by exclusively relying on frozen intermediate outputs and limiting the
  exhaustive exploration of prior knowledge from pre-trained models. Moreover,
  the dependency and redundancy between cross-layer features are frequently
  overlooked, thereby submerging more discriminative representations and causing
  an inherent performance gap (vs. conventional PETL methods). Hence, we propose
  an innovative METL strategy called SHERL for resource-limited scenarios to
  decouple the entire adaptation into two successive and complementary processes.
  In the early route, intermediate outputs are consolidated via an
  anti-redundancy operation, enhancing their compatibility for subsequent
  interactions; thereby in the late route, utilizing minimal late pre-trained
  layers could alleviate the peak demand on memory overhead and regulate these
  fairly flexible features into more adaptive and powerful representations for
  new domains. Extensive ablations on vision-and-language and language-only tasks
  show that SHERL combines the strengths of both parameter and memory-efficient
  techniques, performing on-par or better across diverse architectures with lower
  memory during fine-tuning. Our code is publicly available at:
  https://github.com/Paranioar/SHERL.
paper_authors._id: 668f489bb93b425b7da8978b
paper_authors.name: Haiwen Diao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T08:25:52.287Z
paper_authors._id: 64b4a717aa03b6520839e9b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
paper_authors.fullname: Haiwen Diao
paper_authors.type: user
paper_authors.user: Paranioar
paper_authors._id: 668f489bb93b425b7da8978c
paper_authors.name: Bo Wan
paper_authors._id: 668f489bb93b425b7da8978d
paper_authors.name: Xu Jia
paper_authors._id: 668f489bb93b425b7da8978e
paper_authors.name: Yunzhi Zhuge
paper_authors._id: 668f489bb93b425b7da8978f
paper_authors.name: Ying Zhang
paper_authors._id: 668f489bb93b425b7da89790
paper_authors.name: Huchuan Lu
paper_authors._id: 668f489bb93b425b7da89791
paper_authors.name: Long Chen
paper_title: SHERL: Synthesizing High Accuracy and Efficient Memory for
    Resource-Limited Transfer Learning
paper_summary: Parameter-efficient transfer learning (PETL) has emerged as a flourishing
  research field for adapting large pre-trained models to downstream tasks,
  greatly reducing trainable parameters while grappling with memory challenges
  during fine-tuning. To address it, memory-efficient series (METL) avoid
  backpropagating gradients through the large backbone. However, they compromise
  by exclusively relying on frozen intermediate outputs and limiting the
  exhaustive exploration of prior knowledge from pre-trained models. Moreover,
  the dependency and redundancy between cross-layer features are frequently
  overlooked, thereby submerging more discriminative representations and causing
  an inherent performance gap (vs. conventional PETL methods). Hence, we propose
  an innovative METL strategy called SHERL for resource-limited scenarios to
  decouple the entire adaptation into two successive and complementary processes.
  In the early route, intermediate outputs are consolidated via an
  anti-redundancy operation, enhancing their compatibility for subsequent
  interactions; thereby in the late route, utilizing minimal late pre-trained
  layers could alleviate the peak demand on memory overhead and regulate these
  fairly flexible features into more adaptive and powerful representations for
  new domains. Extensive ablations on vision-and-language and language-only tasks
  show that SHERL combines the strengths of both parameter and memory-efficient
  techniques, performing on-par or better across diverse architectures with lower
  memory during fine-tuning. Our code is publicly available at:
  https://github.com/Paranioar/SHERL.
paper_ai_keywords: PETL (Parameter-efficient transfer learning) METL (memory-efficient series) backpropagating gradients frozen intermediate outputs prior knowledge cross-layer features discriminative representations performance gap SHERL anti-redundancy operation memory overhead flexible features adaptive and powerful representations vision-and-language tasks language-only tasks ablations