title: Can Large Language Models Understand Context?
summary: Understanding context is key to understanding human language, an ability
  which Large Language Models (LLMs) have been increasingly seen to demonstrate
  to an impressive extent. However, though the evaluation of LLMs encompasses
  various domains within the realm of Natural Language Processing, limited
  attention has been paid to probing their linguistic capability of understanding
  contextual features. This paper introduces a context understanding benchmark by
  adapting existing datasets to suit the evaluation of generative models. This
  benchmark comprises of four distinct tasks and nine datasets, all featuring
  prompts designed to assess the models' ability to understand context. First, we
  evaluate the performance of LLMs under the in-context learning pretraining
  scenario. Experimental results indicate that pre-trained dense models struggle
  with understanding more nuanced contextual features when compared to
  state-of-the-art fine-tuned models. Second, as LLM compression holds growing
  significance in both research and real-world applications, we assess the
  context understanding of quantized models under in-context-learning settings.
  We find that 3-bit post-training quantization leads to varying degrees of
  performance reduction on our benchmark. We conduct an extensive analysis of
  these scenarios to substantiate our experimental results.
paper_authors._id: 65bc6cec358734fd096bc182
paper_authors.name: Yilun Zhu
paper_authors._id: 65bc6cec358734fd096bc183
paper_authors.name: Joel Ruben Antony Moniz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:25:22.272Z
paper_authors._id: 6650e6b9e29125c00a919a4d
paper_authors.avatarUrl: /avatars/b0093fa9da51e42168545e83750c3e81.svg
paper_authors.fullname: Joel Moniz
paper_authors.type: user
paper_authors.user: joelmoniz
paper_authors._id: 65bc6cec358734fd096bc184
paper_authors.name: Shruti Bhargava
paper_authors._id: 65bc6cec358734fd096bc185
paper_authors.name: Jiarui Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:36:16.850Z
paper_authors._id: 64bad07938953777fe3ed5e9
paper_authors.avatarUrl: /avatars/91c07a176d8d51111873751e82a4e0ea.svg
paper_authors.fullname: Jiarui Lu
paper_authors.type: user
paper_authors.user: jiarui-lu
paper_authors._id: 65bc6cec358734fd096bc186
paper_authors.name: Dhivya Piraviperumal
paper_authors._id: 65bc6cec358734fd096bc187
paper_authors.name: Site Li
paper_authors._id: 65bc6cec358734fd096bc188
paper_authors.name: Yuan Zhang
paper_authors._id: 65bc6cec358734fd096bc189
paper_authors.name: Hong Yu
paper_authors._id: 65bc6cec358734fd096bc18a
paper_authors.name: Bo-Hsiang Tseng
paper_title: Can Large Language Models Understand Context?
paper_summary: Understanding context is key to understanding human language, an ability
  which Large Language Models (LLMs) have been increasingly seen to demonstrate
  to an impressive extent. However, though the evaluation of LLMs encompasses
  various domains within the realm of Natural Language Processing, limited
  attention has been paid to probing their linguistic capability of understanding
  contextual features. This paper introduces a context understanding benchmark by
  adapting existing datasets to suit the evaluation of generative models. This
  benchmark comprises of four distinct tasks and nine datasets, all featuring
  prompts designed to assess the models' ability to understand context. First, we
  evaluate the performance of LLMs under the in-context learning pretraining
  scenario. Experimental results indicate that pre-trained dense models struggle
  with understanding more nuanced contextual features when compared to
  state-of-the-art fine-tuned models. Second, as LLM compression holds growing
  significance in both research and real-world applications, we assess the
  context understanding of quantized models under in-context-learning settings.
  We find that 3-bit post-training quantization leads to varying degrees of
  performance reduction on our benchmark. We conduct an extensive analysis of
  these scenarios to substantiate our experimental results.
paper_ai_summary: The benchmark evaluates LLMs' context understanding by assessing pre-trained and quantized models across four tasks and nine datasets, highlighting the performance differences between dense and fine-tuned models.
paper_ai_keywords: Large Language Models (LLMs) in-context learning pretraining fine-tuning contextual features prompt design benchmark 3-bit post-training quantization