Same Task, More Tokens: the Impact of Input Length on the Reasoning
  Performance of Large Language Models
This paper explores the impact of extending input lengths on the capabilities
of Large Language Models (LLMs). Despite LLMs advancements in recent times,
their performance consistency across different input lengths is not well
understood. We investigate this aspect by introducing a novel QA reasoning
framework, specifically designed to assess the impact of input length. We
isolate the effect of input length using multiple versions of the same sample,
each being extended with padding of different lengths, types and locations. Our
findings show a notable degradation in LLMs' reasoning performance at much
shorter input lengths than their technical maximum. We show that the
degradation trend appears in every version of our dataset, although at
different intensities. Additionally, our study reveals that traditional
perplexity metrics do not correlate with performance of LLMs' in long input
reasoning tasks. We analyse our results and identify failure modes that can
serve as useful guides for future research, potentially informing strategies to
address the limitations observed in LLMs.
65dbff0c6b8ab39009d47193
Mosh Levy
admin_assigned
2024-02-26T10:26:40.051Z
622f35a2bc2a392eaf21b3e7
/avatars/383409ebd912ba90d8e7966e61a3910d.svg
Mosh Levy
user
Mosh
65dbff0c6b8ab39009d47194
Alon Jacoby
admin_assigned
2024-02-26T10:26:47.978Z
635010c84a9fc01602f5bb8b
/avatars/60c9ff03ae069967b3cda9e69700bee9.svg
Alon Jacoby
user
alonj
65dbff0c6b8ab39009d47195
Yoav Goldberg
claimed_verified
2024-12-30T19:41:36.357Z
676d5323dd95830fd94d675f
/avatars/a8622e2a189a13c7b5a99177c3a87c37.svg
Yoav Goldberg
user
yoavgoldberg
Same Task, More Tokens: the Impact of Input Length on the Reasoning
  Performance of Large Language Models
This paper explores the impact of extending input lengths on the capabilities
of Large Language Models (LLMs). Despite LLMs advancements in recent times,
their performance consistency across different input lengths is not well
understood. We investigate this aspect by introducing a novel QA reasoning
framework, specifically designed to assess the impact of input length. We
isolate the effect of input length using multiple versions of the same sample,
each being extended with padding of different lengths, types and locations. Our
findings show a notable degradation in LLMs' reasoning performance at much
shorter input lengths than their technical maximum. We show that the
degradation trend appears in every version of our dataset, although at
different intensities. Additionally, our study reveals that traditional
perplexity metrics do not correlate with performance of LLMs' in long input
reasoning tasks. We analyse our results and identify failure modes that can
serve as useful guides for future research, potentially informing strategies to
address the limitations observed in LLMs.
Studies the impact of varying input lengths on Large Language Models, revealing performance degradation at shorter lengths and introducing a QA reasoning framework to assess these effects.
Large Language Models QA reasoning framework padding perplexity metrics