Can Large Language Models Infer Causal Relationships from Real-World
  Text?
Understanding and inferring causal relationships from texts is a core aspect
of human cognition and is essential for advancing large language models (LLMs)
towards artificial general intelligence. Existing work primarily focuses on
synthetically generated texts which involve simple causal relationships
explicitly mentioned in the text. This fails to reflect the complexities of
real-world tasks. In this paper, we investigate whether LLMs are capable of
inferring causal relationships from real-world texts. We develop a benchmark
drawn from real-world academic literature which includes diverse texts with
respect to length, complexity of relationships (different levels of
explicitness, number of events, and causal relationships), and domains and
sub-domains. To the best of our knowledge, our benchmark is the first-ever
real-world dataset for this task. Our experiments on state-of-the-art LLMs
evaluated on our proposed benchmark demonstrate significant challenges, with
the best-performing model achieving an average F1 score of only 0.477. Analysis
reveals common pitfalls: difficulty with implicitly stated information, in
distinguishing relevant causal factors from surrounding contextual details, and
with connecting causally relevant information spread across lengthy textual
passages. By systematically characterizing these deficiencies, our benchmark
offers targeted insights for further research into advancing LLM causal
reasoning.
6838c9f0402ef780fdb23293
Ryan Saklad
6838c9f0402ef780fdb23294
Aman Chadha
claimed_verified
2025-05-30T06:54:41.457Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
6838c9f0402ef780fdb23295
Oleg Pavlov
6838c9f0402ef780fdb23296
Raha Moraffah
Can Large Language Models Infer Causal Relationships from Real-World
  Text?
Understanding and inferring causal relationships from texts is a core aspect
of human cognition and is essential for advancing large language models (LLMs)
towards artificial general intelligence. Existing work primarily focuses on
synthetically generated texts which involve simple causal relationships
explicitly mentioned in the text. This fails to reflect the complexities of
real-world tasks. In this paper, we investigate whether LLMs are capable of
inferring causal relationships from real-world texts. We develop a benchmark
drawn from real-world academic literature which includes diverse texts with
respect to length, complexity of relationships (different levels of
explicitness, number of events, and causal relationships), and domains and
sub-domains. To the best of our knowledge, our benchmark is the first-ever
real-world dataset for this task. Our experiments on state-of-the-art LLMs
evaluated on our proposed benchmark demonstrate significant challenges, with
the best-performing model achieving an average F1 score of only 0.477. Analysis
reveals common pitfalls: difficulty with implicitly stated information, in
distinguishing relevant causal factors from surrounding contextual details, and
with connecting causally relevant information spread across lengthy textual
passages. By systematically characterizing these deficiencies, our benchmark
offers targeted insights for further research into advancing LLM causal
reasoning.
A benchmark for assessing LLMs' ability to infer causal relationships from real-world texts highlights significant challenges, revealing common pitfalls in handling implicit information and long-range connections.
large language models causal relationships real-world texts benchmark F1 score causal reasoning contextual details long-range connections