SHERL: Synthesizing High Accuracy and Efficient Memory for
  Resource-Limited Transfer Learning
Parameter-efficient transfer learning (PETL) has emerged as a flourishing
research field for adapting large pre-trained models to downstream tasks,
greatly reducing trainable parameters while grappling with memory challenges
during fine-tuning. To address it, memory-efficient series (METL) avoid
backpropagating gradients through the large backbone. However, they compromise
by exclusively relying on frozen intermediate outputs and limiting the
exhaustive exploration of prior knowledge from pre-trained models. Moreover,
the dependency and redundancy between cross-layer features are frequently
overlooked, thereby submerging more discriminative representations and causing
an inherent performance gap (vs. conventional PETL methods). Hence, we propose
an innovative METL strategy called SHERL for resource-limited scenarios to
decouple the entire adaptation into two successive and complementary processes.
In the early route, intermediate outputs are consolidated via an
anti-redundancy operation, enhancing their compatibility for subsequent
interactions; thereby in the late route, utilizing minimal late pre-trained
layers could alleviate the peak demand on memory overhead and regulate these
fairly flexible features into more adaptive and powerful representations for
new domains. Extensive ablations on vision-and-language and language-only tasks
show that SHERL combines the strengths of both parameter and memory-efficient
techniques, performing on-par or better across diverse architectures with lower
memory during fine-tuning. Our code is publicly available at:
https://github.com/Paranioar/SHERL.
668f489bb93b425b7da8978b
Haiwen Diao
claimed_verified
2024-07-11T08:25:52.287Z
64b4a717aa03b6520839e9b8
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
Haiwen Diao
user
Paranioar
668f489bb93b425b7da8978c
Bo Wan
668f489bb93b425b7da8978d
Xu Jia
668f489bb93b425b7da8978e
Yunzhi Zhuge
668f489bb93b425b7da8978f
Ying Zhang
668f489bb93b425b7da89790
Huchuan Lu
668f489bb93b425b7da89791
Long Chen
SHERL: Synthesizing High Accuracy and Efficient Memory for
  Resource-Limited Transfer Learning
Parameter-efficient transfer learning (PETL) has emerged as a flourishing
research field for adapting large pre-trained models to downstream tasks,
greatly reducing trainable parameters while grappling with memory challenges
during fine-tuning. To address it, memory-efficient series (METL) avoid
backpropagating gradients through the large backbone. However, they compromise
by exclusively relying on frozen intermediate outputs and limiting the
exhaustive exploration of prior knowledge from pre-trained models. Moreover,
the dependency and redundancy between cross-layer features are frequently
overlooked, thereby submerging more discriminative representations and causing
an inherent performance gap (vs. conventional PETL methods). Hence, we propose
an innovative METL strategy called SHERL for resource-limited scenarios to
decouple the entire adaptation into two successive and complementary processes.
In the early route, intermediate outputs are consolidated via an
anti-redundancy operation, enhancing their compatibility for subsequent
interactions; thereby in the late route, utilizing minimal late pre-trained
layers could alleviate the peak demand on memory overhead and regulate these
fairly flexible features into more adaptive and powerful representations for
new domains. Extensive ablations on vision-and-language and language-only tasks
show that SHERL combines the strengths of both parameter and memory-efficient
techniques, performing on-par or better across diverse architectures with lower
memory during fine-tuning. Our code is publicly available at:
https://github.com/Paranioar/SHERL.
SHERL, an innovative METL strategy, decouples model adaptation into early and late routes to balance parameter and memory efficiency while maintaining or enhancing performance across vision-and-language and language-only tasks.
parameter-efficient transfer learning PETL memory-efficient series METL backpropagation intermediate outputs cross-layer features discriminative representations resource-limited scenarios anti-redundancy operation late pre-trained layers memory overhead adaptive representations