title: Can Large Language Models Infer Causation from Correlation?
summary: Causal inference is one of the hallmarks of human intelligence. While the
  field of CausalNLP has attracted much interest in the recent years, existing
  causal inference datasets in NLP primarily rely on discovering causality from
  empirical knowledge (e.g., commonsense knowledge). In this work, we propose the
  first benchmark dataset to test the pure causal inference skills of large
  language models (LLMs). Specifically, we formulate a novel task Corr2Cause,
  which takes a set of correlational statements and determines the causal
  relationship between the variables. We curate a large-scale dataset of more
  than 400K samples, on which we evaluate seventeen existing LLMs. Through our
  experiments, we identify a key shortcoming of LLMs in terms of their causal
  inference skills, and show that these models achieve almost close to random
  performance on the task. This shortcoming is somewhat mitigated when we try to
  re-purpose LLMs for this skill via finetuning, but we find that these models
  still fail to generalize -- they can only perform causal inference in
  in-distribution settings when variable names and textual expressions used in
  the queries are similar to those in the training set, but fail in
  out-of-distribution settings generated by perturbing these queries. Corr2Cause
  is a challenging task for LLMs, and would be helpful in guiding future research
  on improving LLMs' pure reasoning skills and generalizability. Our data is at
  https://huggingface.co/datasets/causalnlp/corr2cause. Our code is at
  https://github.com/causalNLP/corr2cause.
paper_authors._id: 64866f2ba4cf2081f20f7ecd
paper_authors.name: Zhijing Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T11:57:11.555Z
paper_authors._id: 646a524d3eb2bab0419b0653
paper_authors.avatarUrl: /avatars/56c468d4083e16a0660a51965d2cd842.svg
paper_authors.fullname: Causal NLP
paper_authors.type: user
paper_authors.user: causal-nlp
paper_authors._id: 64866f2ba4cf2081f20f7ece
paper_authors.name: Jiarui Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-12T18:59:11.555Z
paper_authors._id: 64706ed2fa9fd77212de57ed
paper_authors.avatarUrl: /avatars/e27564d1ba5bd9ac2f228a9c8e4c9506.svg
paper_authors.fullname: Jiarui Liu
paper_authors.type: user
paper_authors.user: Jerry999
paper_authors._id: 64866f2ba4cf2081f20f7ecf
paper_authors.name: Zhiheng Lyu
paper_authors._id: 64866f2ba4cf2081f20f7ed0
paper_authors.name: Spencer Poff
paper_authors._id: 64866f2ba4cf2081f20f7ed1
paper_authors.name: Mrinmaya Sachan
paper_authors._id: 64866f2ba4cf2081f20f7ed2
paper_authors.name: Rada Mihalcea
paper_authors._id: 64866f2ba4cf2081f20f7ed3
paper_authors.name: Mona Diab
paper_authors._id: 64866f2ba4cf2081f20f7ed4
paper_authors.name: Bernhard Schölkopf
paper_title: Can Large Language Models Infer Causation from Correlation?
paper_summary: Causal inference is one of the hallmarks of human intelligence. While the
  field of CausalNLP has attracted much interest in the recent years, existing
  causal inference datasets in NLP primarily rely on discovering causality from
  empirical knowledge (e.g., commonsense knowledge). In this work, we propose the
  first benchmark dataset to test the pure causal inference skills of large
  language models (LLMs). Specifically, we formulate a novel task Corr2Cause,
  which takes a set of correlational statements and determines the causal
  relationship between the variables. We curate a large-scale dataset of more
  than 400K samples, on which we evaluate seventeen existing LLMs. Through our
  experiments, we identify a key shortcoming of LLMs in terms of their causal
  inference skills, and show that these models achieve almost close to random
  performance on the task. This shortcoming is somewhat mitigated when we try to
  re-purpose LLMs for this skill via finetuning, but we find that these models
  still fail to generalize -- they can only perform causal inference in
  in-distribution settings when variable names and textual expressions used in
  the queries are similar to those in the training set, but fail in
  out-of-distribution settings generated by perturbing these queries. Corr2Cause
  is a challenging task for LLMs, and would be helpful in guiding future research
  on improving LLMs' pure reasoning skills and generalizability. Our data is at
  https://huggingface.co/datasets/causalnlp/corr2cause. Our code is at
  https://github.com/causalNLP/corr2cause.
paper_ai_summary: A benchmark dataset Corr2Cause tests the causal inference skills of large language models, revealing their limitations and highlighting the need for improved generalizability.
paper_ai_keywords: causal inference CausalNLP large language models Corr2Cause empirical knowledge commonsense knowledge causal relationship finetuning in-distribution out-of-distribution