The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic
  Competence in Large Language Models
Current large language models (LLMs) have demonstrated emerging capabilities
in social intelligence tasks, including implicature resolution (Sravanthi et
al. (2024)) and theory-of-mind reasoning (Shapira et al. (2024)), both of which
require substantial pragmatic understanding. However, how LLMs acquire this
competence throughout the training process remains poorly understood. In this
work, we introduce ALTPRAG, a dataset grounded in the pragmatic concept of
alternatives, designed to evaluate whether LLMs at different training stages
can accurately infer nuanced speaker intentions. Each instance pairs two
contextually appropriate but pragmatically distinct continuations, enabling
fine-grained assessment of both pragmatic interpretation and contrastive
reasoning. We systematically evaluate 22 LLMs across key training stages:
pre-training, supervised fine-tuning (SFT), and preference optimization, to
examine the development of pragmatic competence. Our results show that even
base models exhibit notable sensitivity to pragmatic cues, which improves
consistently with increases in model and data scale. Additionally, SFT and RLHF
contribute further gains, particularly in cognitive-pragmatic reasoning. These
findings highlight pragmatic competence as an emergent and compositional
property of LLM training and offer new insights for aligning models with human
communicative norms.
68365b399951c3c658a05903
Kefan Yu
68365b399951c3c658a05904
Qingcheng Zeng
68365b399951c3c658a05905
Weihao Xuan
claimed_verified
2025-07-22T07:54:35.861Z
65b8909c89eb3dfbe8d26780
https://cdn-avatars.huggingface.co/v1/production/uploads/65b8909c89eb3dfbe8d26780/aJExXsCN9oQOdd9bsQOEO.jpeg
Weihao XUAN
user
weihao1115
68365b399951c3c658a05906
Wanxin Li
68365b399951c3c658a05907
Jingyi Wu
68365b399951c3c658a05908
Rob Voigt
The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic
  Competence in Large Language Models
Current large language models (LLMs) have demonstrated emerging capabilities
in social intelligence tasks, including implicature resolution (Sravanthi et
al. (2024)) and theory-of-mind reasoning (Shapira et al. (2024)), both of which
require substantial pragmatic understanding. However, how LLMs acquire this
competence throughout the training process remains poorly understood. In this
work, we introduce ALTPRAG, a dataset grounded in the pragmatic concept of
alternatives, designed to evaluate whether LLMs at different training stages
can accurately infer nuanced speaker intentions. Each instance pairs two
contextually appropriate but pragmatically distinct continuations, enabling
fine-grained assessment of both pragmatic interpretation and contrastive
reasoning. We systematically evaluate 22 LLMs across key training stages:
pre-training, supervised fine-tuning (SFT), and preference optimization, to
examine the development of pragmatic competence. Our results show that even
base models exhibit notable sensitivity to pragmatic cues, which improves
consistently with increases in model and data scale. Additionally, SFT and RLHF
contribute further gains, particularly in cognitive-pragmatic reasoning. These
findings highlight pragmatic competence as an emergent and compositional
property of LLM training and offer new insights for aligning models with human
communicative norms.
ALTPRAG dataset evaluates pragmatic competence in LLMs across training stages, showing improvements with increased scale and specific training methods.
LLMs social intelligence tasks implicature resolution theory-of-mind reasoning pragmatic understanding alternative pragmatic concept pragmatic interpretation contrastive reasoning pre-training supervised fine-tuning preference optimization cognitive-pragmatic reasoning RLHF pragmatic cues emergent property compositional property