HARP: Hesitation-Aware Reframing in Transformer Inference Pass
This paper aims to improve the performance of large language models by
addressing the variable computational demands in inference steps, where some
tokens require more computational resources than others. We present HARP, a
simple modification to "off-the-shelf" Transformer forward pass. Drawing from
hesitation and the framing effect in decision-making, HARP selectively applies
additional computation when the model encounters uncertainty during token
generation. Our method mimics human cognitive processes by pausing at difficult
decision points and reframing inputs for a different perspective. Unlike other
approaches, HARP is model-agnostic, training-free, and easy to implement. We
thoroughly evaluate our method across various downstream tasks and model sizes,
demonstrating performance improvements up to +5.16%. Notably, HARP achieves
these gains while maintaining inference times twice faster than beam search.
Simple and yet with significant gains, HARP offers a practical solution for
enhancing the performance of Transformer-based language models with minimal
computational impact.
67595e5f6621cef8bdbde626
Romain Storaï
claimed_verified
2024-12-11T09:54:16.623Z
631f1d51c1a8269da3941374
https://cdn-avatars.huggingface.co/v1/production/uploads/631f1d51c1a8269da3941374/53__nPiD9t0e1qzrY0DlD.png
Romain Storaï
user
romsto
67595e5f6621cef8bdbde627
Seung-won Hwang
HARP: Hesitation-Aware Reframing in Transformer Inference Pass
This paper aims to improve the performance of large language models by
addressing the variable computational demands in inference steps, where some
tokens require more computational resources than others. We present HARP, a
simple modification to "off-the-shelf" Transformer forward pass. Drawing from
hesitation and the framing effect in decision-making, HARP selectively applies
additional computation when the model encounters uncertainty during token
generation. Our method mimics human cognitive processes by pausing at difficult
decision points and reframing inputs for a different perspective. Unlike other
approaches, HARP is model-agnostic, training-free, and easy to implement. We
thoroughly evaluate our method across various downstream tasks and model sizes,
demonstrating performance improvements up to +5.16%. Notably, HARP achieves
these gains while maintaining inference times twice faster than beam search.
Simple and yet with significant gains, HARP offers a practical solution for
enhancing the performance of Transformer-based language models with minimal
computational impact.
HARP, a model-agnostic modification to the Transformer forward pass, enhances language model performance by selectively applying additional computation during uncertain token generation, achieving significant improvements with faster inference times.
Transformers HARP model-agnostic token generation uncertainty beam search