Long-context LLMs Struggle with Long In-context Learning
Large Language Models (LLMs) have made significant strides in handling long
sequences exceeding 32K tokens. However, their performance evaluation has
largely been confined to metrics like perplexity and synthetic tasks, which may
not fully capture their abilities in more nuanced, real-world scenarios. This
study introduces a specialized benchmark (LIConBench) focusing on long
in-context learning within the realm of extreme-label classification. We
meticulously selected six datasets with a label range spanning 28 to 174
classes covering different input (few-shot demonstration) length from 2K to
50K. Our benchmark requires LLMs to comprehend the entire input to recognize
the massive label spaces to make correct prediction. We evaluate 13
long-context LLMs on our benchmarks. We find that the long-context LLMs perform
relatively well under the token length of 20K and the performance benefits from
utilizing the long context window. However, after the context window exceeds
20K, most LLMs except GPT-4 will dip dramatically. This suggests a notable gap
in current LLM capabilities for processing and understanding long, context-rich
sequences. Further analysis revealed a tendency among models to favor
predictions for labels presented towards the end at the sequence. Their ability
to reason over multiple pieces in the long sequence is yet to be improved. Our
study reveals that long context understanding and reasoning is still a
challenging task for the existing LLMs. We believe LIConBench could serve as a
more realistic evaluation for the future long context LLMs.
660cbbb57411be24fb6116ae
Tianle Li
660cbbb57411be24fb6116af
Ge Zhang
claimed_verified
2024-04-05T10:05:20.245Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
660cbbb57411be24fb6116b0
Quy Duc Do
claimed_verified
2025-05-27T13:57:53.779Z
65fa5fa1180d06d37c845d51
/avatars/a04eb59f0a3a3b5800ae443a8eaab785.svg
Quy Duc Do
user
qd2do
660cbbb57411be24fb6116b1
Xiang Yue
claimed_verified
2024-05-27T07:21:12.271Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
660cbbb57411be24fb6116b2
Wenhu Chen
Long-context LLMs Struggle with Long In-context Learning
Large Language Models (LLMs) have made significant strides in handling long
sequences exceeding 32K tokens. However, their performance evaluation has
largely been confined to metrics like perplexity and synthetic tasks, which may
not fully capture their abilities in more nuanced, real-world scenarios. This
study introduces a specialized benchmark (LIConBench) focusing on long
in-context learning within the realm of extreme-label classification. We
meticulously selected six datasets with a label range spanning 28 to 174
classes covering different input (few-shot demonstration) length from 2K to
50K. Our benchmark requires LLMs to comprehend the entire input to recognize
the massive label spaces to make correct prediction. We evaluate 13
long-context LLMs on our benchmarks. We find that the long-context LLMs perform
relatively well under the token length of 20K and the performance benefits from
utilizing the long context window. However, after the context window exceeds
20K, most LLMs except GPT-4 will dip dramatically. This suggests a notable gap
in current LLM capabilities for processing and understanding long, context-rich
sequences. Further analysis revealed a tendency among models to favor
predictions for labels presented towards the end at the sequence. Their ability
to reason over multiple pieces in the long sequence is yet to be improved. Our
study reveals that long context understanding and reasoning is still a
challenging task for the existing LLMs. We believe LIConBench could serve as a
more realistic evaluation for the future long context LLMs.
LIConBench evaluates long-context LLMs on extreme-label classification tasks with sequences up to 50K tokens, highlighting performance dips beyond 20K tokens and favoring of recent labels.
Large Language Models long in-context learning extreme-label classification LIConBench long-context LLMs long context understanding reasoning