Mixed-precision Supernet Training from Vision Foundation Models using Low Rank Adapter
<|reference_start|>Mixed-precision Supernet Training from Vision Foundation Models using Low Rank Adapter: Compression of large and performant vision foundation models (VFMs) into arbitrary bit-wise operations (BitOPs) allows their deployment on various hardware. We propose to fine-tune a VFM to a mixed-precision quantized supernet. The supernet-based neural architecture search (NAS) can be adopted for this purpose, which trains a supernet, and then subnets within arbitrary hardware budgets can be extracted. However, existing methods face difficulties in optimizing the mixed-precision search space and incurring large memory costs during training. To tackle these challenges, first, we study the effective search space design for fine-tuning a VFM by comparing different operators (such as resolution, feature size, width, depth, and bit-widths) in terms of performance and BitOPs reduction. Second, we propose memory-efficient supernet training using a low-rank adapter (LoRA) and a progressive training strategy. The proposed method is evaluated for the recently proposed VFM, Segment Anything Model, fine-tuned on segmentation tasks. The searched model yields about a 95% reduction in BitOPs without incurring performance degradation.<|reference_end|>
@article{sakuma2024mixed-precision,
    title={Mixed-precision Supernet Training from Vision Foundation Models using
  Low Rank Adapter},
    author={Yuiko Sakuma, Masakazu Yoshimura, Junji Otsuka, Atsushi Irie, Takeshi
  Ohashi},
    journal={arXiv preprint arXiv:2403.20080},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.20080},
    primaryClass={cs.CV}
}