Measuring Faithfulness in Chain-of-Thought Reasoning
Large language models (LLMs) perform better when they produce step-by-step,
"Chain-of-Thought" (CoT) reasoning before answering a question, but it is
unclear if the stated reasoning is a faithful explanation of the model's actual
reasoning (i.e., its process for answering the question). We investigate
hypotheses for how CoT reasoning may be unfaithful, by examining how the model
predictions change when we intervene on the CoT (e.g., by adding mistakes or
paraphrasing it). Models show large variation across tasks in how strongly they
condition on the CoT when predicting their answer, sometimes relying heavily on
the CoT and other times primarily ignoring it. CoT's performance boost does not
seem to come from CoT's added test-time compute alone or from information
encoded via the particular phrasing of the CoT. As models become larger and
more capable, they produce less faithful reasoning on most tasks we study.
Overall, our results suggest that CoT can be faithful if the circumstances such
as the model size and task are carefully chosen.
64c1c4697e6975b3916e6ddd
Tamera Lanham
64c1c4697e6975b3916e6dde
Anna Chen
64c1c4697e6975b3916e6ddf
Ansh Radhakrishnan
admin_assigned
2023-07-27T08:57:42.888Z
625e0426e1176e6c772aa37c
/avatars/a05c5e3b964ca2a6b82eec0e3d89c064.svg
Ansh Radhakrishnan
user
anshr
64c1c4697e6975b3916e6de0
Benoit Steiner
64c1c4697e6975b3916e6de1
Carson Denison
64c1c4697e6975b3916e6de2
Danny Hernandez
admin_assigned
2023-07-27T08:58:16.063Z
63fe9f9452441fe3e878b631
/avatars/de5cf5dc8b7e9d8d0dfef24850a4d4b3.svg
Danny Hernandez
user
DannyDev06
64c1c4697e6975b3916e6de3
Dustin Li
64c1c4697e6975b3916e6de4
Esin Durmus
admin_assigned
2023-07-27T08:58:53.907Z
627dd7dbba19b19eff6aaa46
/avatars/6e43025ccb96e48339d1db477da043b0.svg
Esin Durmus
user
esdurmus
64c1c4697e6975b3916e6de5
Evan Hubinger
64c1c4697e6975b3916e6de6
Jackson Kernion
64c1c4697e6975b3916e6de7
Kamilė Lukošiūtė
64c1c4697e6975b3916e6de8
Karina Nguyen
admin_assigned
2023-07-27T08:59:15.889Z
62ef4469cbe9a53fcd5a7f03
/avatars/13352152a67ef304ba444cbc310be5dd.svg
Karina Nguyen
user
karinanguyen
64c1c4697e6975b3916e6de9
Newton Cheng
admin_assigned
2023-07-27T08:59:24.513Z
6230f73441ca6e8d8f614008
/avatars/2c6f84c6ecb71c047c5d043d5a89f791.svg
Newton Cheng
user
nccheng
64c1c4697e6975b3916e6dea
Nicholas Joseph
64c1c4697e6975b3916e6deb
Nicholas Schiefer
admin_assigned
2023-07-27T08:59:54.082Z
63a246e2931d2c13ad842a59
/avatars/9ac4f3d01c293ce97507be402dfa688c.svg
Nicholas Schiefer
user
nschiefer
64c1c4697e6975b3916e6dec
Oliver Rausch
64c1c4697e6975b3916e6ded
Robin Larson
64c1c4697e6975b3916e6dee
Sam McCandlish
64c1c4697e6975b3916e6def
Sandipan Kundu
64c1c4697e6975b3916e6df0
Saurav Kadavath
64c1c4697e6975b3916e6df1
Shannon Yang
admin_assigned
2023-07-27T09:00:45.948Z
62ce2249342b1d5dab9451f4
/avatars/9a5f6992b7842ecf721c6908245d75ba.svg
Shannon Yang
user
shannonyang
64c1c4697e6975b3916e6df2
Thomas Henighan
64c1c4697e6975b3916e6df3
Timothy Maxwell
64c1c4697e6975b3916e6df4
Timothy Telleen-Lawton
64c1c4697e6975b3916e6df5
Tristan Hume
64c1c4697e6975b3916e6df6
Zac Hatfield-Dodds
64c1c4697e6975b3916e6df7
Jared Kaplan
admin_assigned
2023-07-27T10:39:11.814Z
633b051c7af633cbcd01b0ff
/avatars/e2f315a8d6241ae1a783ba2957a211a5.svg
Jared Kaplan
user
FrizzleFried
64c1c4697e6975b3916e6df8
Jan Brauner
64c1c4697e6975b3916e6df9
Samuel R. Bowman
64c1c4697e6975b3916e6dfa
Ethan Perez
extracted_pending
2023-07-27T01:12:09.591Z
62bb28dc4d3faf9c8938f742
/avatars/cceeb6b8e679b7460519024cd5e301b5.svg
Ethan
user
ethanjperez
Measuring Faithfulness in Chain-of-Thought Reasoning
Large language models (LLMs) perform better when they produce step-by-step,
"Chain-of-Thought" (CoT) reasoning before answering a question, but it is
unclear if the stated reasoning is a faithful explanation of the model's actual
reasoning (i.e., its process for answering the question). We investigate
hypotheses for how CoT reasoning may be unfaithful, by examining how the model
predictions change when we intervene on the CoT (e.g., by adding mistakes or
paraphrasing it). Models show large variation across tasks in how strongly they
condition on the CoT when predicting their answer, sometimes relying heavily on
the CoT and other times primarily ignoring it. CoT's performance boost does not
seem to come from CoT's added test-time compute alone or from information
encoded via the particular phrasing of the CoT. As models become larger and
more capable, they produce less faithful reasoning on most tasks we study.
Overall, our results suggest that CoT can be faithful if the circumstances such
as the model size and task are carefully chosen.
Larger language models may not produce faithful reasoning when using chain-of-thought despite performance improvements, depending on model size and task.
chain-of-thought CoT reasoning model predictions intervention test-time compute