This paper presents a comprehensive and structured study of
PEFT methods for PLMs. By classifying the PEFT methods in
NLP, we identify the main techniques and challenges associ-
ated with them. We employ several representative PEFT meth-
ods to fine-tune encoder-based RoBERTa, encoder-decoder-
based T5, and decoder-based LLaMA on various downstream
tasks. Experimental results reveal that most PEFT methods
significantly improve parameter efficiency and achieve com-
parable or even better performance compared to full fine-
tuning. Additionally, most PEFT methods lower the memory
footprint, with QLoRA drastically reducing the computational
memory requirement, and alleviating the memory challenge
when fine-tuning LLMs. Furthermore, we introduce common
applications of PEFT methods and outline future research
directions. As the development of LLMs continues, there is
a clear need to develop PEFT methods that can effectively
reduce computational resource demands and memory usage
Parameter-EfficientFine-TuningMethods