MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model
<|reference_start|>MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model: The emergence of foundation models, such as the Segment Anything Model (SAM), has sparked interest in Parameter-Efficient Fine-Tuning (PEFT) methods that tailor these large models to application domains outside their training data. However, different PEFT techniques modify the representation of a model differently, making it a non-trivial task to select the most appropriate method for the domain of interest. We propose a new framework, Mixture-of-PEFTs methods (MoPEFT), that is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.<|reference_end|>
@article{sahay2024mopeft:,
    title={MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model},
    author={Rajat Sahay, Andreas Savakis},
    journal={arXiv preprint arXiv:2405.00293},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.00293},
    primaryClass={cs.CV}
}