As the name suggests, in the paradigm of full-parameter ﬁne-tuning, all model param-
eters are subject to change during training. With a higher training cost than the
PEFT, full-parameter ﬁne-tuning can generally lead to better performance than the
parameter-eﬃcient methods [108]. However, this phenomenon may not hold true on a
simple dataset (e.g., a dataset with a lack of language diversity) for a speciﬁc down-
stream task [109]. Since PEFT only trains a relatively small number of parameters,
models trained via full-parameter ﬁne-tuning methods have a greater learning capacity.
Moreover, the convergence of PEFT is generally not as fast as that of full-parameter
ﬁne-tuning [110]. As for training cost, it is reported [108] that “using full-parameters
ﬁne-tuning requires about 3-5 times the time cost” of LoRA ﬁne-tuning. GPU memory
consumption is also a concern because updating all the parameters can be impracti-
Beyond Efficiency