Frontier Language Models are not Robust to Adversarial Arithmetic, or
  "What do I need to say so you agree 2+2=5?
We introduce and study the problem of adversarial arithmetic, which provides
a simple yet challenging testbed for language model alignment. This problem is
comprised of arithmetic questions posed in natural language, with an arbitrary
adversarial string inserted before the question is complete. Even in the simple
setting of 1-digit addition problems, it is easy to find adversarial prompts
that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and
even to steer models to a particular wrong answer. We additionally provide a
simple algorithm for finding successful attacks by querying those same models,
which we name "prompt inversion rejection sampling" (PIRS). We finally show
that models can be partially hardened against these attacks via reinforcement
learning and via agentic constitutional loops. However, we were not able to
make a language model fully robust against adversarial arithmetic attacks.
65542745854adee4f53825ef
C. Daniel Freeman
65542745854adee4f53825f0
Laura Culp
65542745854adee4f53825f1
Aaron Parisi
admin_assigned
2023-11-15T10:23:00.127Z
635f493ba81c7f7424a702ae
/avatars/6f05d4f76609607966ed4635cb1e0a02.svg
Aaron Parisi
user
AnotherA
65542745854adee4f53825f2
Maxwell L Bileschi
65542745854adee4f53825f3
Gamaleldin F Elsayed
65542745854adee4f53825f4
Alex Rizkowsky
65542745854adee4f53825f5
Isabelle Simpson
65542745854adee4f53825f6
Alex Alemi
admin_assigned
2023-11-15T10:23:26.141Z
6303ff84fc783bfc74441afc
/avatars/bc295e0d94c9c8be0c6a8264587ad4a5.svg
Alex Alemi
user
knowflake
65542745854adee4f53825f7
Azade Nova
65542745854adee4f53825f8
Ben Adlam
65542745854adee4f53825f9
Bernd Bohnet
65542745854adee4f53825fa
Gaurav Mishra
admin_assigned
2023-11-15T10:23:52.890Z
6275324958ea49d11062e0e0
/avatars/8db608697460da2ec391de8d7981cd78.svg
Gaurav Mishra
user
gauravmishra
65542745854adee4f53825fb
Hanie Sedghi
65542745854adee4f53825fc
Igor Mordatch
65542745854adee4f53825fd
Izzeddin Gur
admin_assigned
2023-11-15T10:24:09.553Z
64c0ccc19832a455bc2a2069
/avatars/9d4fdf13cd62c6c258b5f50d67c8a1b1.svg
IZZEDDIN GUR
user
izzeddingur
65542745854adee4f53825fe
Jaehoon Lee
65542745854adee4f53825ff
JD Co-Reyes
65542745854adee4f5382600
Jeffrey Pennington
65542745854adee4f5382601
Kelvin Xu
65542745854adee4f5382602
Kevin Swersky
admin_assigned
2023-11-15T10:25:17.053Z
630e6ef664f1f8d0c771b758
/avatars/1df2bfadb2b6fdf8307189936efc6ef0.svg
Kevin Swersky
user
kswersky
65542745854adee4f5382603
Kshiteej Mahajan
65542745854adee4f5382604
Lechao Xiao
65542745854adee4f5382605
Rosanne Liu
65542745854adee4f5382606
Simon Kornblith
admin_assigned
2023-11-15T10:25:44.761Z
6443ffc2af034cdfd69c5c00
/avatars/0c12cf9baaf3b24a02ffb42238cdc6df.svg
Simon Kornblith
user
simonsterk
65542745854adee4f5382607
Noah Constant
65542745854adee4f5382608
Peter J. Liu
admin_assigned
2023-11-15T10:25:57.777Z
62ded294dcdc9043efaa4fc3
/avatars/1294078b8cd5c1901e4021253d877357.svg
Peter J Liu
user
peterjliu
65542745854adee4f5382609
Roman Novak
admin_assigned
2023-11-15T10:26:08.790Z
63e2769bf0740bec2bfff1a1
/avatars/e2841a836319d43035bcddfbfe3344d0.svg
Roman Novak
user
kulpims
65542745854adee4f538260a
Sharad Vikram
65542745854adee4f538260b
Yundi Qian
65542745854adee4f538260c
Noah Fiedel
admin_assigned
2023-11-15T10:26:29.855Z
62f82ebd870a3f98bbf9e62a
/avatars/3b03299e128abf2c8ba82db704855de7.svg
Noah Fiedel
user
nfiedel
65542745854adee4f538260d
Jascha Sohl-Dickstein
admin_assigned
2023-11-15T10:26:37.575Z
63d209ae8eaa4831006b3b0d
/avatars/51cc0f6dfab2233bddd73bfcd0c8aee3.svg
Jascha Sohl-Dickstein
user
jaschasd
Frontier Language Models are not Robust to Adversarial Arithmetic, or
  "What do I need to say so you agree 2+2=5?
We introduce and study the problem of adversarial arithmetic, which provides
a simple yet challenging testbed for language model alignment. This problem is
comprised of arithmetic questions posed in natural language, with an arbitrary
adversarial string inserted before the question is complete. Even in the simple
setting of 1-digit addition problems, it is easy to find adversarial prompts
that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and
even to steer models to a particular wrong answer. We additionally provide a
simple algorithm for finding successful attacks by querying those same models,
which we name "prompt inversion rejection sampling" (PIRS). We finally show
that models can be partially hardened against these attacks via reinforcement
learning and via agentic constitutional loops. However, we were not able to
make a language model fully robust against adversarial arithmetic attacks.
Adversarial arithmetic, a natural language-based arithmetic challenge with adversarial strings, demonstrates the vulnerability of language models, which can be partially mitigated by reinforcement learning and constitutional loops.
adversarial arithmetic language model alignment prompt inversion rejection sampling PIERS reinforcement learning agentic constitutional loops