Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test
  Generation: An Empirical Study
The advent of large language models (LLMs) like GitHub Copilot has
significantly enhanced programmers' productivity, particularly in code
generation. However, these models often struggle with real-world tasks without
fine-tuning. As LLMs grow larger and more performant, fine-tuning for
specialized tasks becomes increasingly expensive. Parameter-efficient
fine-tuning (PEFT) methods, which fine-tune only a subset of model parameters,
offer a promising solution by reducing the computational costs of tuning LLMs
while maintaining their performance. Existing studies have explored using PEFT
and LLMs for various code-related tasks and found that the effectiveness of
PEFT techniques is task-dependent. The application of PEFT techniques in unit
test generation remains underexplored. The state-of-the-art is limited to using
LLMs with full fine-tuning to generate unit tests. This paper investigates both
full fine-tuning and various PEFT methods, including LoRA, (IA)^3, and prompt
tuning, across different model architectures and sizes. We use well-established
benchmark datasets to evaluate their effectiveness in unit test generation. Our
findings show that PEFT methods can deliver performance comparable to full
fine-tuning for unit test generation, making specialized fine-tuning more
accessible and cost-effective. Notably, prompt tuning is the most effective in
terms of cost and resource utilization, while LoRA approaches the effectiveness
of full fine-tuning in several cases.
673171234edd034cb4bdaf73
André Storhaug
extracted_confirmed
2024-11-11T04:17:16.529Z
621997b88e4346d1e0f23bc9
https://cdn-avatars.huggingface.co/v1/production/uploads/621997b88e4346d1e0f23bc9/jPnWYzo0ZDbHfSjny11j_.png
André Storhaug
user
andstor
673171234edd034cb4bdaf74
Jingyue Li
Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test
  Generation: An Empirical Study
The advent of large language models (LLMs) like GitHub Copilot has
significantly enhanced programmers' productivity, particularly in code
generation. However, these models often struggle with real-world tasks without
fine-tuning. As LLMs grow larger and more performant, fine-tuning for
specialized tasks becomes increasingly expensive. Parameter-efficient
fine-tuning (PEFT) methods, which fine-tune only a subset of model parameters,
offer a promising solution by reducing the computational costs of tuning LLMs
while maintaining their performance. Existing studies have explored using PEFT
and LLMs for various code-related tasks and found that the effectiveness of
PEFT techniques is task-dependent. The application of PEFT techniques in unit
test generation remains underexplored. The state-of-the-art is limited to using
LLMs with full fine-tuning to generate unit tests. This paper investigates both
full fine-tuning and various PEFT methods, including LoRA, (IA)^3, and prompt
tuning, across different model architectures and sizes. We use well-established
benchmark datasets to evaluate their effectiveness in unit test generation. Our
findings show that PEFT methods can deliver performance comparable to full
fine-tuning for unit test generation, making specialized fine-tuning more
accessible and cost-effective. Notably, prompt tuning is the most effective in
terms of cost and resource utilization, while LoRA approaches the effectiveness
of full fine-tuning in several cases.
Research investigates the effectiveness of parameter-efficient fine-tuning methods compared to full fine-tuning for generating unit tests using large language models.
large language models LLMs GitHub Copilot fine-tuning parameter-efficient fine-tuning PEFT LoRA (IA)^3 prompt tuning unit test generation benchmark datasets