title: HARP: Hesitation-Aware Reframing in Transformer Inference Pass
summary: This paper aims to improve the performance of large language models by
  addressing the variable computational demands in inference steps, where some
  tokens require more computational resources than others. We present HARP, a
  simple modification to "off-the-shelf" Transformer forward pass. Drawing from
  hesitation and the framing effect in decision-making, HARP selectively applies
  additional computation when the model encounters uncertainty during token
  generation. Our method mimics human cognitive processes by pausing at difficult
  decision points and reframing inputs for a different perspective. Unlike other
  approaches, HARP is model-agnostic, training-free, and easy to implement. We
  thoroughly evaluate our method across various downstream tasks and model sizes,
  demonstrating performance improvements up to +5.16%. Notably, HARP achieves
  these gains while maintaining inference times twice faster than beam search.
  Simple and yet with significant gains, HARP offers a practical solution for
  enhancing the performance of Transformer-based language models with minimal
  computational impact.
paper_authors._id: 67595e5f6621cef8bdbde626
paper_authors.name: Romain Storaï
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:54:16.623Z
paper_authors._id: 631f1d51c1a8269da3941374
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631f1d51c1a8269da3941374/53__nPiD9t0e1qzrY0DlD.png
paper_authors.fullname: Romain Storaï
paper_authors.type: user
paper_authors.user: romsto
paper_authors._id: 67595e5f6621cef8bdbde627
paper_authors.name: Seung-won Hwang
paper_title: HARP: Hesitation-Aware Reframing in Transformer Inference Pass
paper_summary: This paper aims to improve the performance of large language models by
  addressing the variable computational demands in inference steps, where some
  tokens require more computational resources than others. We present HARP, a
  simple modification to "off-the-shelf" Transformer forward pass. Drawing from
  hesitation and the framing effect in decision-making, HARP selectively applies
  additional computation when the model encounters uncertainty during token
  generation. Our method mimics human cognitive processes by pausing at difficult
  decision points and reframing inputs for a different perspective. Unlike other
  approaches, HARP is model-agnostic, training-free, and easy to implement. We
  thoroughly evaluate our method across various downstream tasks and model sizes,
  demonstrating performance improvements up to +5.16%. Notably, HARP achieves
  these gains while maintaining inference times twice faster than beam search.
  Simple and yet with significant gains, HARP offers a practical solution for
  enhancing the performance of Transformer-based language models with minimal
  computational impact.
paper_ai_keywords: Transformer forward pass HARP (Human-Aware Recursive Processing) hesitation framing effect uncertainty token generation model-agnostic training-free downstream tasks inference times beam search computational impact