title: PAFT: Prompt-Agnostic Fine-Tuning
summary: While Large Language Models (LLMs) adapt well to downstream tasks after
  fine-tuning, this adaptability often compromises prompt robustness, as even
  minor prompt variations can significantly degrade performance. To address this,
  we propose Prompt-Agnostic Fine-Tuning(PAFT), a simple yet effective approach
  that dynamically adjusts prompts during fine-tuning. This encourages the model
  to learn underlying task principles rather than overfitting to specific prompt
  formulations. PAFT operates in two stages: First, a diverse set of meaningful,
  synthetic candidate prompts is constructed. Second, during fine-tuning, prompts
  are randomly sampled from this set to create dynamic training inputs. Extensive
  experiments across diverse datasets and LLMs demonstrate that models trained
  with PAFT exhibit strong robustness and generalization across a wide range of
  prompts, including unseen ones. This enhanced robustness improves both model
  performance and inference speed while maintaining training efficiency. Ablation
  studies further confirm the effectiveness of PAFT.
paper_authors._id: 67b576aa489d68b981e086ad
paper_authors.name: Chenxing Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T10:23:00.016Z
paper_authors._id: 65ed3051492a7f35db21fea2
paper_authors.avatarUrl: /avatars/4fc0ccc21aa88e4e8ff74a6f850570b8.svg
paper_authors.fullname: Chenxing Wei
paper_authors.type: user
paper_authors.user: kittttttt
paper_authors._id: 67b576aa489d68b981e086ae
paper_authors.name: Yao Shu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T14:37:34.696Z
paper_authors._id: 66123816d7dfcea8ae55a751
paper_authors.avatarUrl: /avatars/3f24468b63e4babd7d9a0c926ca01b23.svg
paper_authors.fullname: Shu Yao
paper_authors.type: user
paper_authors.user: ZCODE0
paper_authors._id: 67b576aa489d68b981e086af
paper_authors.name: Mingwen Ou
paper_authors._id: 67b576aa489d68b981e086b0
paper_authors.name: Ying Tiffany He
paper_authors._id: 67b576aa489d68b981e086b1
paper_authors.name: Fei Richard Yu
paper_title: PAFT: Prompt-Agnostic Fine-Tuning
paper_summary: While Large Language Models (LLMs) adapt well to downstream tasks after
  fine-tuning, this adaptability often compromises prompt robustness, as even
  minor prompt variations can significantly degrade performance. To address this,
  we propose Prompt-Agnostic Fine-Tuning(PAFT), a simple yet effective approach
  that dynamically adjusts prompts during fine-tuning. This encourages the model
  to learn underlying task principles rather than overfitting to specific prompt
  formulations. PAFT operates in two stages: First, a diverse set of meaningful,
  synthetic candidate prompts is constructed. Second, during fine-tuning, prompts
  are randomly sampled from this set to create dynamic training inputs. Extensive
  experiments across diverse datasets and LLMs demonstrate that models trained
  with PAFT exhibit strong robustness and generalization across a wide range of
  prompts, including unseen ones. This enhanced robustness improves both model
  performance and inference speed while maintaining training efficiency. Ablation
  studies further confirm the effectiveness of PAFT.
paper_ai_summary: Prompt-Agnostic Fine-Tuning (PAFT) dynamically adjusts prompts during training to improve LLM robustness and generalization.
paper_ai_keywords: Large Language Models (LLMs) fine-tuning prompt robustness Prompt-Agnostic Fine-Tuning (PAFT) synthetic prompt dynamic training inputs