title: Astraios: Parameter-Efficient Instruction Tuning Code Large Language
    Models
summary: The high cost of full-parameter fine-tuning (FFT) of Large Language Models
  (LLMs) has led to a series of parameter-efficient fine-tuning (PEFT) methods.
  However, it remains unclear which methods provide the best cost-performance
  trade-off at different model scales. We introduce Astraios, a suite of 28
  instruction-tuned OctoCoder models using 7 tuning methods and 4 model sizes up
  to 16 billion parameters. Through investigations across 5 tasks and 8 different
  datasets encompassing both code comprehension and code generation tasks, we
  find that FFT generally leads to the best downstream performance across all
  scales, and PEFT methods differ significantly in their efficacy based on the
  model scale. LoRA usually offers the most favorable trade-off between cost and
  performance. Further investigation into the effects of these methods on both
  model robustness and code security reveals that larger models tend to
  demonstrate reduced robustness and less security. At last, we explore the
  relationships among updated parameters, cross-entropy loss, and task
  performance. We find that the tuning effectiveness observed in small models
  generalizes well to larger models, and the validation loss in instruction
  tuning can be a reliable indicator of overall downstream performance.
paper_authors._id: 659393950f4519bfc2023451
paper_authors.name: Terry Yue Zhuo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-02T09:26:07.633Z
paper_authors._id: 62b7fb545233925f253531c8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
paper_authors.fullname: Terry Yue Zhuo
paper_authors.type: user
paper_authors.user: terryyz
paper_authors._id: 659393950f4519bfc2023452
paper_authors.name: Armel Zebaze
paper_authors._id: 659393950f4519bfc2023453
paper_authors.name: Nitchakarn Suppattarachai
paper_authors._id: 659393950f4519bfc2023454
paper_authors.name: Leandro von Werra
paper_authors._id: 659393950f4519bfc2023455
paper_authors.name: Harm de Vries
paper_authors._id: 659393950f4519bfc2023456
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-02T09:26:03.828Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 659393950f4519bfc2023457
paper_authors.name: Niklas Muennighoff
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-08T08:48:12.083Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_title: Astraios: Parameter-Efficient Instruction Tuning Code Large Language
    Models
paper_summary: The high cost of full-parameter fine-tuning (FFT) of Large Language Models
  (LLMs) has led to a series of parameter-efficient fine-tuning (PEFT) methods.
  However, it remains unclear which methods provide the best cost-performance
  trade-off at different model scales. We introduce Astraios, a suite of 28
  instruction-tuned OctoCoder models using 7 tuning methods and 4 model sizes up
  to 16 billion parameters. Through investigations across 5 tasks and 8 different
  datasets encompassing both code comprehension and code generation tasks, we
  find that FFT generally leads to the best downstream performance across all
  scales, and PEFT methods differ significantly in their efficacy based on the
  model scale. LoRA usually offers the most favorable trade-off between cost and
  performance. Further investigation into the effects of these methods on both
  model robustness and code security reveals that larger models tend to
  demonstrate reduced robustness and less security. At last, we explore the
  relationships among updated parameters, cross-entropy loss, and task
  performance. We find that the tuning effectiveness observed in small models
  generalizes well to larger models, and the validation loss in instruction
  tuning can be a reliable indicator of overall downstream performance.
paper_ai_keywords: full-parameter fine-tuning (FFT) Large Language Models (LLMs) parameter-efficient fine-tuning (PEFT) Astraios instruction-tuned OctoCoder model scales code comprehension code generation LoRA model robustness code security cross-entropy loss task performance validation loss instruction tuning