Can Large Language Models Infer Causation from Correlation?
Causal inference is one of the hallmarks of human intelligence. While the
field of CausalNLP has attracted much interest in the recent years, existing
causal inference datasets in NLP primarily rely on discovering causality from
empirical knowledge (e.g., commonsense knowledge). In this work, we propose the
first benchmark dataset to test the pure causal inference skills of large
language models (LLMs). Specifically, we formulate a novel task Corr2Cause,
which takes a set of correlational statements and determines the causal
relationship between the variables. We curate a large-scale dataset of more
than 400K samples, on which we evaluate seventeen existing LLMs. Through our
experiments, we identify a key shortcoming of LLMs in terms of their causal
inference skills, and show that these models achieve almost close to random
performance on the task. This shortcoming is somewhat mitigated when we try to
re-purpose LLMs for this skill via finetuning, but we find that these models
still fail to generalize -- they can only perform causal inference in
in-distribution settings when variable names and textual expressions used in
the queries are similar to those in the training set, but fail in
out-of-distribution settings generated by perturbing these queries. Corr2Cause
is a challenging task for LLMs, and would be helpful in guiding future research
on improving LLMs' pure reasoning skills and generalizability. Our data is at
https://huggingface.co/datasets/causalnlp/corr2cause. Our code is at
https://github.com/causalNLP/corr2cause.
64866f2ba4cf2081f20f7ecd
Zhijing Jin
admin_assigned
2023-06-12T11:57:11.555Z
646a524d3eb2bab0419b0653
/avatars/56c468d4083e16a0660a51965d2cd842.svg
Causal NLP
user
causal-nlp
64866f2ba4cf2081f20f7ece
Jiarui Liu
claimed_verified
2023-06-12T18:59:11.555Z
64706ed2fa9fd77212de57ed
/avatars/e27564d1ba5bd9ac2f228a9c8e4c9506.svg
Jiarui Liu
user
Jerry999
64866f2ba4cf2081f20f7ecf
Zhiheng Lyu
64866f2ba4cf2081f20f7ed0
Spencer Poff
64866f2ba4cf2081f20f7ed1
Mrinmaya Sachan
64866f2ba4cf2081f20f7ed2
Rada Mihalcea
64866f2ba4cf2081f20f7ed3
Mona Diab
64866f2ba4cf2081f20f7ed4
Bernhard Schölkopf
Can Large Language Models Infer Causation from Correlation?
Causal inference is one of the hallmarks of human intelligence. While the
field of CausalNLP has attracted much interest in the recent years, existing
causal inference datasets in NLP primarily rely on discovering causality from
empirical knowledge (e.g., commonsense knowledge). In this work, we propose the
first benchmark dataset to test the pure causal inference skills of large
language models (LLMs). Specifically, we formulate a novel task Corr2Cause,
which takes a set of correlational statements and determines the causal
relationship between the variables. We curate a large-scale dataset of more
than 400K samples, on which we evaluate seventeen existing LLMs. Through our
experiments, we identify a key shortcoming of LLMs in terms of their causal
inference skills, and show that these models achieve almost close to random
performance on the task. This shortcoming is somewhat mitigated when we try to
re-purpose LLMs for this skill via finetuning, but we find that these models
still fail to generalize -- they can only perform causal inference in
in-distribution settings when variable names and textual expressions used in
the queries are similar to those in the training set, but fail in
out-of-distribution settings generated by perturbing these queries. Corr2Cause
is a challenging task for LLMs, and would be helpful in guiding future research
on improving LLMs' pure reasoning skills and generalizability. Our data is at
https://huggingface.co/datasets/causalnlp/corr2cause. Our code is at
https://github.com/causalNLP/corr2cause.
A benchmark dataset Corr2Cause tests the causal inference skills of large language models, revealing their limitations and highlighting the need for improved generalizability.
causal inference CausalNLP large language models Corr2Cause empirical knowledge commonsense knowledge causal relationship finetuning in-distribution out-of-distribution