title: Long-context LLMs Struggle with Long In-context Learning
summary: Large Language Models (LLMs) have made significant strides in handling long
  sequences exceeding 32K tokens. However, their performance evaluation has
  largely been confined to metrics like perplexity and synthetic tasks, which may
  not fully capture their abilities in more nuanced, real-world scenarios. This
  study introduces a specialized benchmark (LIConBench) focusing on long
  in-context learning within the realm of extreme-label classification. We
  meticulously selected six datasets with a label range spanning 28 to 174
  classes covering different input (few-shot demonstration) length from 2K to
  50K. Our benchmark requires LLMs to comprehend the entire input to recognize
  the massive label spaces to make correct prediction. We evaluate 13
  long-context LLMs on our benchmarks. We find that the long-context LLMs perform
  relatively well under the token length of 20K and the performance benefits from
  utilizing the long context window. However, after the context window exceeds
  20K, most LLMs except GPT-4 will dip dramatically. This suggests a notable gap
  in current LLM capabilities for processing and understanding long, context-rich
  sequences. Further analysis revealed a tendency among models to favor
  predictions for labels presented towards the end at the sequence. Their ability
  to reason over multiple pieces in the long sequence is yet to be improved. Our
  study reveals that long context understanding and reasoning is still a
  challenging task for the existing LLMs. We believe LIConBench could serve as a
  more realistic evaluation for the future long context LLMs.
paper_authors._id: 660cbbb57411be24fb6116ae
paper_authors.name: Tianle Li
paper_authors._id: 660cbbb57411be24fb6116af
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-05T10:05:20.245Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 660cbbb57411be24fb6116b0
paper_authors.name: Quy Duc Do
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T13:57:53.779Z
paper_authors._id: 65fa5fa1180d06d37c845d51
paper_authors.avatarUrl: /avatars/a04eb59f0a3a3b5800ae443a8eaab785.svg
paper_authors.fullname: Quy Duc Do
paper_authors.type: user
paper_authors.user: qd2do
paper_authors._id: 660cbbb57411be24fb6116b1
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T07:21:12.271Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 660cbbb57411be24fb6116b2
paper_authors.name: Wenhu Chen
paper_title: Long-context LLMs Struggle with Long In-context Learning
paper_summary: Large Language Models (LLMs) have made significant strides in handling long
  sequences exceeding 32K tokens. However, their performance evaluation has
  largely been confined to metrics like perplexity and synthetic tasks, which may
  not fully capture their abilities in more nuanced, real-world scenarios. This
  study introduces a specialized benchmark (LIConBench) focusing on long
  in-context learning within the realm of extreme-label classification. We
  meticulously selected six datasets with a label range spanning 28 to 174
  classes covering different input (few-shot demonstration) length from 2K to
  50K. Our benchmark requires LLMs to comprehend the entire input to recognize
  the massive label spaces to make correct prediction. We evaluate 13
  long-context LLMs on our benchmarks. We find that the long-context LLMs perform
  relatively well under the token length of 20K and the performance benefits from
  utilizing the long context window. However, after the context window exceeds
  20K, most LLMs except GPT-4 will dip dramatically. This suggests a notable gap
  in current LLM capabilities for processing and understanding long, context-rich
  sequences. Further analysis revealed a tendency among models to favor
  predictions for labels presented towards the end at the sequence. Their ability
  to reason over multiple pieces in the long sequence is yet to be improved. Our
  study reveals that long context understanding and reasoning is still a
  challenging task for the existing LLMs. We believe LIConBench could serve as a
  more realistic evaluation for the future long context LLMs.
paper_ai_summary: LIConBench evaluates long-context LLMs on extreme-label classification tasks with sequences up to 50K tokens, highlighting performance dips beyond 20K tokens and favoring of recent labels.
paper_ai_keywords: Large Language Models long in-context learning extreme-label classification LIConBench long-context LLMs long context understanding reasoning