Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration
<|reference_start|>Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration: Recently, pre-trained model and efficient parameter tuning have achieved remarkable success in natural language processing and high-level computer vision with the aid of masked modeling and prompt tuning. In low-level computer vision, however, there have been limited investigations on pre-trained models and even efficient fine-tuning strategy has not yet been explored despite its importance and benefit in various real-world tasks such as alleviating memory inflation issue when integrating new tasks on AI edge devices. Here, we propose a novel efficient parameter tuning approach dubbed contribution-based low-rank adaptation (CoLoRA) for multiple image restorations along with effective pre-training method with random order degradations (PROD). Unlike prior arts that tune all network parameters, our CoLoRA effectively fine-tunes small amount of parameters by leveraging LoRA (low-rank adaptation) for each new vision task with our contribution-based method to adaptively determine layer by layer capacity for that task to yield comparable performance to full tuning. Furthermore, our PROD strategy allows to extend the capability of pre-trained models with improved performance as well as robustness to bridge synthetic pre-training and real-world fine-tuning. Our CoLoRA with PROD has demonstrated its superior performance in various image restoration tasks across diverse degradation types on both synthetic and real-world datasets for known and novel tasks.<|reference_end|>
@article{park2024contribution-based,
    title={Contribution-based Low-Rank Adaptation with Pre-training Model for Real
  Image Restoration},
    author={Donwon Park, Hayeon Kim, Se Young Chun},
    journal={arXiv preprint arXiv:2408.01099},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.01099},
    primaryClass={cs.CV cs.AI}
}