title: Same Task, More Tokens: the Impact of Input Length on the Reasoning
    Performance of Large Language Models
summary: This paper explores the impact of extending input lengths on the capabilities
  of Large Language Models (LLMs). Despite LLMs advancements in recent times,
  their performance consistency across different input lengths is not well
  understood. We investigate this aspect by introducing a novel QA reasoning
  framework, specifically designed to assess the impact of input length. We
  isolate the effect of input length using multiple versions of the same sample,
  each being extended with padding of different lengths, types and locations. Our
  findings show a notable degradation in LLMs' reasoning performance at much
  shorter input lengths than their technical maximum. We show that the
  degradation trend appears in every version of our dataset, although at
  different intensities. Additionally, our study reveals that traditional
  perplexity metrics do not correlate with performance of LLMs' in long input
  reasoning tasks. We analyse our results and identify failure modes that can
  serve as useful guides for future research, potentially informing strategies to
  address the limitations observed in LLMs.
paper_authors._id: 65dbff0c6b8ab39009d47193
paper_authors.name: Mosh Levy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:26:40.051Z
paper_authors._id: 622f35a2bc2a392eaf21b3e7
paper_authors.avatarUrl: /avatars/383409ebd912ba90d8e7966e61a3910d.svg
paper_authors.fullname: Mosh Levy
paper_authors.type: user
paper_authors.user: Mosh
paper_authors._id: 65dbff0c6b8ab39009d47194
paper_authors.name: Alon Jacoby
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:26:47.978Z
paper_authors._id: 635010c84a9fc01602f5bb8b
paper_authors.avatarUrl: /avatars/60c9ff03ae069967b3cda9e69700bee9.svg
paper_authors.fullname: Alon Jacoby
paper_authors.type: user
paper_authors.user: alonj
paper_authors._id: 65dbff0c6b8ab39009d47195
paper_authors.name: Yoav Goldberg
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:41:36.357Z
paper_authors._id: 676d5323dd95830fd94d675f
paper_authors.avatarUrl: /avatars/a8622e2a189a13c7b5a99177c3a87c37.svg
paper_authors.fullname: Yoav Goldberg
paper_authors.type: user
paper_authors.user: yoavgoldberg
paper_title: Same Task, More Tokens: the Impact of Input Length on the Reasoning
    Performance of Large Language Models
paper_summary: This paper explores the impact of extending input lengths on the capabilities
  of Large Language Models (LLMs). Despite LLMs advancements in recent times,
  their performance consistency across different input lengths is not well
  understood. We investigate this aspect by introducing a novel QA reasoning
  framework, specifically designed to assess the impact of input length. We
  isolate the effect of input length using multiple versions of the same sample,
  each being extended with padding of different lengths, types and locations. Our
  findings show a notable degradation in LLMs' reasoning performance at much
  shorter input lengths than their technical maximum. We show that the
  degradation trend appears in every version of our dataset, although at
  different intensities. Additionally, our study reveals that traditional
  perplexity metrics do not correlate with performance of LLMs' in long input
  reasoning tasks. We analyse our results and identify failure modes that can
  serve as useful guides for future research, potentially informing strategies to
  address the limitations observed in LLMs.
paper_ai_keywords: Large Language Models (LLMs) QA reasoning framework input length padding reasoning performance perplexity metrics