title: Attention Overflow: Language Model Input Blur during Long-Context
    Missing Items Recommendation
summary: Large language models (LLMs) can suggest missing elements from items listed
  in a prompt, which can be used for list completion or recommendations based on
  users' history. However, their performance degrades when presented with too
  many items, as they start to suggest items already included in the input list.
  This occurs at around 100 items for mid-2024 flagship LLMs. We evaluate this
  phenomenon on both synthetic problems (e.g., finding missing numbers in a given
  range of shuffled integers) and realistic movie recommendation scenarios. We
  refer to this issue as attention overflow, as preventing repetition
  requires attending to all items simultaneously. Although iterative loops can
  mitigate this problem, their costs increase with the repetition rate, affecting
  the language models' ability to derive novelty from lengthy inputs.
paper_authors._id: 669a125fac60504c1348eb8e
paper_authors.name: Damien Sileo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:22:53.551Z
paper_authors._id: 5fc0bcb41160c47d1d43856b
paper_authors.avatarUrl: /avatars/4d68767d363f7bc5605906c0e994f132.svg
paper_authors.fullname: Damien Sileo
paper_authors.type: user
paper_authors.user: sileod
paper_title: Attention Overflow: Language Model Input Blur during Long-Context
    Missing Items Recommendation
paper_summary: Large language models (LLMs) can suggest missing elements from items listed
  in a prompt, which can be used for list completion or recommendations based on
  users' history. However, their performance degrades when presented with too
  many items, as they start to suggest items already included in the input list.
  This occurs at around 100 items for mid-2024 flagship LLMs. We evaluate this
  phenomenon on both synthetic problems (e.g., finding missing numbers in a given
  range of shuffled integers) and realistic movie recommendation scenarios. We
  refer to this issue as attention overflow, as preventing repetition
  requires attending to all items simultaneously. Although iterative loops can
  mitigate this problem, their costs increase with the repetition rate, affecting
  the language models' ability to derive novelty from lengthy inputs.
paper_ai_keywords: large language models (LLMs) list completion recommendations users' history performance degradation synthetic problems finding missing numbers shuffled integers movie recommendation scenarios attention overflow iterative loops novelty