title: Can Large Language Models Infer Causal Relationships from Real-World
    Text?
summary: Understanding and inferring causal relationships from texts is a core aspect
  of human cognition and is essential for advancing large language models (LLMs)
  towards artificial general intelligence. Existing work primarily focuses on
  synthetically generated texts which involve simple causal relationships
  explicitly mentioned in the text. This fails to reflect the complexities of
  real-world tasks. In this paper, we investigate whether LLMs are capable of
  inferring causal relationships from real-world texts. We develop a benchmark
  drawn from real-world academic literature which includes diverse texts with
  respect to length, complexity of relationships (different levels of
  explicitness, number of events, and causal relationships), and domains and
  sub-domains. To the best of our knowledge, our benchmark is the first-ever
  real-world dataset for this task. Our experiments on state-of-the-art LLMs
  evaluated on our proposed benchmark demonstrate significant challenges, with
  the best-performing model achieving an average F1 score of only 0.477. Analysis
  reveals common pitfalls: difficulty with implicitly stated information, in
  distinguishing relevant causal factors from surrounding contextual details, and
  with connecting causally relevant information spread across lengthy textual
  passages. By systematically characterizing these deficiencies, our benchmark
  offers targeted insights for further research into advancing LLM causal
  reasoning.
paper_authors._id: 6838c9f0402ef780fdb23293
paper_authors.name: Ryan Saklad
paper_authors._id: 6838c9f0402ef780fdb23294
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:41.457Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_authors._id: 6838c9f0402ef780fdb23295
paper_authors.name: Oleg Pavlov
paper_authors._id: 6838c9f0402ef780fdb23296
paper_authors.name: Raha Moraffah
paper_title: Can Large Language Models Infer Causal Relationships from Real-World
    Text?
paper_summary: Understanding and inferring causal relationships from texts is a core aspect
  of human cognition and is essential for advancing large language models (LLMs)
  towards artificial general intelligence. Existing work primarily focuses on
  synthetically generated texts which involve simple causal relationships
  explicitly mentioned in the text. This fails to reflect the complexities of
  real-world tasks. In this paper, we investigate whether LLMs are capable of
  inferring causal relationships from real-world texts. We develop a benchmark
  drawn from real-world academic literature which includes diverse texts with
  respect to length, complexity of relationships (different levels of
  explicitness, number of events, and causal relationships), and domains and
  sub-domains. To the best of our knowledge, our benchmark is the first-ever
  real-world dataset for this task. Our experiments on state-of-the-art LLMs
  evaluated on our proposed benchmark demonstrate significant challenges, with
  the best-performing model achieving an average F1 score of only 0.477. Analysis
  reveals common pitfalls: difficulty with implicitly stated information, in
  distinguishing relevant causal factors from surrounding contextual details, and
  with connecting causally relevant information spread across lengthy textual
  passages. By systematically characterizing these deficiencies, our benchmark
  offers targeted insights for further research into advancing LLM causal
  reasoning.
paper_ai_summary: A benchmark for assessing LLMs' ability to infer causal relationships from real-world texts highlights significant challenges, revealing common pitfalls in handling implicit information and long-range connections.
paper_ai_keywords: large language models causal relationships real-world texts benchmark F1 score causal reasoning contextual details long-range connections