title: The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic
    Competence in Large Language Models
summary: Current large language models (LLMs) have demonstrated emerging capabilities
  in social intelligence tasks, including implicature resolution (Sravanthi et
  al. (2024)) and theory-of-mind reasoning (Shapira et al. (2024)), both of which
  require substantial pragmatic understanding. However, how LLMs acquire this
  competence throughout the training process remains poorly understood. In this
  work, we introduce ALTPRAG, a dataset grounded in the pragmatic concept of
  alternatives, designed to evaluate whether LLMs at different training stages
  can accurately infer nuanced speaker intentions. Each instance pairs two
  contextually appropriate but pragmatically distinct continuations, enabling
  fine-grained assessment of both pragmatic interpretation and contrastive
  reasoning. We systematically evaluate 22 LLMs across key training stages:
  pre-training, supervised fine-tuning (SFT), and preference optimization, to
  examine the development of pragmatic competence. Our results show that even
  base models exhibit notable sensitivity to pragmatic cues, which improves
  consistently with increases in model and data scale. Additionally, SFT and RLHF
  contribute further gains, particularly in cognitive-pragmatic reasoning. These
  findings highlight pragmatic competence as an emergent and compositional
  property of LLM training and offer new insights for aligning models with human
  communicative norms.
paper_authors._id: 68365b399951c3c658a05903
paper_authors.name: Kefan Yu
paper_authors._id: 68365b399951c3c658a05904
paper_authors.name: Qingcheng Zeng
paper_authors._id: 68365b399951c3c658a05905
paper_authors.name: Weihao Xuan
paper_authors._id: 68365b399951c3c658a05906
paper_authors.name: Wanxin Li
paper_authors._id: 68365b399951c3c658a05907
paper_authors.name: Jingyi Wu
paper_authors._id: 68365b399951c3c658a05908
paper_authors.name: Rob Voigt
paper_title: The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic
    Competence in Large Language Models
paper_summary: Current large language models (LLMs) have demonstrated emerging capabilities
  in social intelligence tasks, including implicature resolution (Sravanthi et
  al. (2024)) and theory-of-mind reasoning (Shapira et al. (2024)), both of which
  require substantial pragmatic understanding. However, how LLMs acquire this
  competence throughout the training process remains poorly understood. In this
  work, we introduce ALTPRAG, a dataset grounded in the pragmatic concept of
  alternatives, designed to evaluate whether LLMs at different training stages
  can accurately infer nuanced speaker intentions. Each instance pairs two
  contextually appropriate but pragmatically distinct continuations, enabling
  fine-grained assessment of both pragmatic interpretation and contrastive
  reasoning. We systematically evaluate 22 LLMs across key training stages:
  pre-training, supervised fine-tuning (SFT), and preference optimization, to
  examine the development of pragmatic competence. Our results show that even
  base models exhibit notable sensitivity to pragmatic cues, which improves
  consistently with increases in model and data scale. Additionally, SFT and RLHF
  contribute further gains, particularly in cognitive-pragmatic reasoning. These
  findings highlight pragmatic competence as an emergent and compositional
  property of LLM training and offer new insights for aligning models with human
  communicative norms.
paper_ai_summary: ALTPRAG dataset evaluates pragmatic competence in LLMs across training stages, showing improvements with increased scale and specific training methods.
paper_ai_keywords: LLMs social intelligence tasks implicature resolution theory-of-mind reasoning pragmatic understanding alternative pragmatic concept pragmatic interpretation contrastive reasoning pre-training supervised fine-tuning preference optimization cognitive-pragmatic reasoning RLHF pragmatic cues emergent property compositional property