SeCoKD: Aligning Large Language Models for In-Context Learning with Fewer Shots
<|reference_start|>SeCoKD: Aligning Large Language Models for In-Context Learning with Fewer Shots: Previous studies have shown that demonstrations can significantly help Large Language Models (LLMs ) perform better on the given tasks. However, this so-called In-Context Learning ( ICL ) ability is very sensitive to the presenting context, and often dozens of demonstrations are needed. In this work, we investigate if we can reduce the shot number while still maintaining a competitive performance. We present SeCoKD, a self-Knowledge Distillation ( KD ) training framework that aligns the student model with a heavily prompted variation, thereby increasing the utilization of a single demonstration. We experiment with the SeCoKD across three LLMs and six benchmarks focusing mainly on reasoning tasks. Results show that our method outperforms the base model and Supervised Fine-tuning ( SFT ), especially in zero-shot and one-shot settings by 30% and 10%, respectively. Moreover, SeCoKD brings little negative artifacts when evaluated on new tasks, which is more robust than Supervised Fine-tuning.<|reference_end|>
@article{wang2024secokd:,
    title={SeCoKD: Aligning Large Language Models for In-Context Learning with
  Fewer Shots},
    author={Weixing Wang, Haojin Yang, Christoph Meinel},
    journal={arXiv preprint arXiv:2406.14208},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.14208},
    primaryClass={cs.AI}
}