Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanations
<|reference_start|>Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanations: Incorporating natural language rationales in the prompt and In-Context Learning (ICL) have led to a significant improvement of Large Language Models (LLMs) performance. However, generating high-quality rationales require human-annotation or the use of auxiliary proxy models. In this work, we propose Self-AMPLIFY to automatically generate rationales from post hoc explanation methods applied to Small Language Models (SLMs) to improve their own performance. Self-AMPLIFY is a 3-step method that targets samples, generates rationales and builds a final prompt to leverage ICL. Self-AMPLIFY performance is evaluated on four SLMs and five datasets requiring strong reasoning abilities. Self-AMPLIFY achieves good results against competitors, leading to strong accuracy improvement. Self-AMPLIFY is the first method to apply post hoc explanation methods to autoregressive language models to generate rationales to improve their own performance in a fully automated manner.<|reference_end|>
@article{bhan2024self-amplify:,
    title={Self-AMPLIFY: Improving Small Language Models with Self Post Hoc
  Explanations},
    author={Milan Bhan and Jean-Noel Vittaut and Nicolas Chesneau and Marie-Jeanne
  Lesot},
    journal={arXiv preprint arXiv:2402.12038},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.12038},
    primaryClass={cs.LG cs.CL}
}