Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision
<|reference_start|>Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision: Parameter-efficient fine-tuning (PEFT) has become increasingly important as foundation models continue to grow in both popularity and size. Adapter has been particularly well-received due to their potential for parameter reduction and adaptability across diverse tasks. However, striking a balance between high efficiency and robust generalization across tasks remains a challenge for adapter-based methods. We analyze existing methods and find that: 1) parameter sharing is the key to reducing redundancy; 2) more tunable parameters, dynamic allocation, and block-specific design are keys to improving performance. Unfortunately, no previous work considers all these factors. Inspired by this insight, we introduce a novel framework named Adapter-X. First, a Sharing Mixture of Adapters (SMoA) module is proposed to fulfill token-level dynamic allocation, increased tunable parameters, and inter-block sharing at the same time. Second, some block-specific designs like Prompt Generator (PG) are introduced to further enhance the ability of adaptation. Extensive experiments across 2D image and 3D point cloud modalities demonstrate that Adapter-X represents a significant milestone as it is the first to outperform full fine-tuning in both 2D image and 3D point cloud modalities with significantly fewer parameters, i.e., only 0.20% and 1.88% of original trainable parameters for 2D and 3D classification tasks. Our code will be publicly available.<|reference_end|>
@article{li2024adapter-x:,
    title={Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for
  Vision},
    author={Minglei Li, Peng Ye, Yongqi Huang, Lin Zhang, Tao Chen, Tong He,
  Jiayuan Fan, Wanli Ouyang},
    journal={arXiv preprint arXiv:2406.03051},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.03051},
    primaryClass={cs.CV}
}