title: Frontier Language Models are not Robust to Adversarial Arithmetic, or
    "What do I need to say so you agree 2+2=5?
summary: We introduce and study the problem of adversarial arithmetic, which provides
  a simple yet challenging testbed for language model alignment. This problem is
  comprised of arithmetic questions posed in natural language, with an arbitrary
  adversarial string inserted before the question is complete. Even in the simple
  setting of 1-digit addition problems, it is easy to find adversarial prompts
  that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and
  even to steer models to a particular wrong answer. We additionally provide a
  simple algorithm for finding successful attacks by querying those same models,
  which we name "prompt inversion rejection sampling" (PIRS). We finally show
  that models can be partially hardened against these attacks via reinforcement
  learning and via agentic constitutional loops. However, we were not able to
  make a language model fully robust against adversarial arithmetic attacks.
paper_authors._id: 65542745854adee4f53825ef
paper_authors.name: C. Daniel Freeman
paper_authors._id: 65542745854adee4f53825f0
paper_authors.name: Laura Culp
paper_authors._id: 65542745854adee4f53825f1
paper_authors.name: Aaron Parisi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:23:00.127Z
paper_authors._id: 635f493ba81c7f7424a702ae
paper_authors.avatarUrl: /avatars/6f05d4f76609607966ed4635cb1e0a02.svg
paper_authors.fullname: Aaron Parisi
paper_authors.type: user
paper_authors.user: AnotherA
paper_authors._id: 65542745854adee4f53825f2
paper_authors.name: Maxwell L Bileschi
paper_authors._id: 65542745854adee4f53825f3
paper_authors.name: Gamaleldin F Elsayed
paper_authors._id: 65542745854adee4f53825f4
paper_authors.name: Alex Rizkowsky
paper_authors._id: 65542745854adee4f53825f5
paper_authors.name: Isabelle Simpson
paper_authors._id: 65542745854adee4f53825f6
paper_authors.name: Alex Alemi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:23:26.141Z
paper_authors._id: 6303ff84fc783bfc74441afc
paper_authors.avatarUrl: /avatars/bc295e0d94c9c8be0c6a8264587ad4a5.svg
paper_authors.fullname: Alex Alemi
paper_authors.type: user
paper_authors.user: knowflake
paper_authors._id: 65542745854adee4f53825f7
paper_authors.name: Azade Nova
paper_authors._id: 65542745854adee4f53825f8
paper_authors.name: Ben Adlam
paper_authors._id: 65542745854adee4f53825f9
paper_authors.name: Bernd Bohnet
paper_authors._id: 65542745854adee4f53825fa
paper_authors.name: Gaurav Mishra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:23:52.890Z
paper_authors._id: 6275324958ea49d11062e0e0
paper_authors.avatarUrl: /avatars/8db608697460da2ec391de8d7981cd78.svg
paper_authors.fullname: Gaurav Mishra
paper_authors.type: user
paper_authors.user: gauravmishra
paper_authors._id: 65542745854adee4f53825fb
paper_authors.name: Hanie Sedghi
paper_authors._id: 65542745854adee4f53825fc
paper_authors.name: Igor Mordatch
paper_authors._id: 65542745854adee4f53825fd
paper_authors.name: Izzeddin Gur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:24:09.553Z
paper_authors._id: 64c0ccc19832a455bc2a2069
paper_authors.avatarUrl: /avatars/9d4fdf13cd62c6c258b5f50d67c8a1b1.svg
paper_authors.fullname: IZZEDDIN GUR
paper_authors.type: user
paper_authors.user: izzeddingur
paper_authors._id: 65542745854adee4f53825fe
paper_authors.name: Jaehoon Lee
paper_authors._id: 65542745854adee4f53825ff
paper_authors.name: JD Co-Reyes
paper_authors._id: 65542745854adee4f5382600
paper_authors.name: Jeffrey Pennington
paper_authors._id: 65542745854adee4f5382601
paper_authors.name: Kelvin Xu
paper_authors._id: 65542745854adee4f5382602
paper_authors.name: Kevin Swersky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:25:17.053Z
paper_authors._id: 630e6ef664f1f8d0c771b758
paper_authors.avatarUrl: /avatars/1df2bfadb2b6fdf8307189936efc6ef0.svg
paper_authors.fullname: Kevin Swersky
paper_authors.type: user
paper_authors.user: kswersky
paper_authors._id: 65542745854adee4f5382603
paper_authors.name: Kshiteej Mahajan
paper_authors._id: 65542745854adee4f5382604
paper_authors.name: Lechao Xiao
paper_authors._id: 65542745854adee4f5382605
paper_authors.name: Rosanne Liu
paper_authors._id: 65542745854adee4f5382606
paper_authors.name: Simon Kornblith
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:25:44.761Z
paper_authors._id: 6443ffc2af034cdfd69c5c00
paper_authors.avatarUrl: /avatars/0c12cf9baaf3b24a02ffb42238cdc6df.svg
paper_authors.fullname: Simon Kornblith
paper_authors.type: user
paper_authors.user: simonsterk
paper_authors._id: 65542745854adee4f5382607
paper_authors.name: Noah Constant
paper_authors._id: 65542745854adee4f5382608
paper_authors.name: Peter J. Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:25:57.777Z
paper_authors._id: 62ded294dcdc9043efaa4fc3
paper_authors.avatarUrl: /avatars/1294078b8cd5c1901e4021253d877357.svg
paper_authors.fullname: Peter J Liu
paper_authors.type: user
paper_authors.user: peterjliu
paper_authors._id: 65542745854adee4f5382609
paper_authors.name: Roman Novak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:26:08.790Z
paper_authors._id: 63e2769bf0740bec2bfff1a1
paper_authors.avatarUrl: /avatars/e2841a836319d43035bcddfbfe3344d0.svg
paper_authors.fullname: Roman Novak
paper_authors.type: user
paper_authors.user: kulpims
paper_authors._id: 65542745854adee4f538260a
paper_authors.name: Sharad Vikram
paper_authors._id: 65542745854adee4f538260b
paper_authors.name: Yundi Qian
paper_authors._id: 65542745854adee4f538260c
paper_authors.name: Noah Fiedel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:26:29.855Z
paper_authors._id: 62f82ebd870a3f98bbf9e62a
paper_authors.avatarUrl: /avatars/3b03299e128abf2c8ba82db704855de7.svg
paper_authors.fullname: Noah Fiedel
paper_authors.type: user
paper_authors.user: nfiedel
paper_authors._id: 65542745854adee4f538260d
paper_authors.name: Jascha Sohl-Dickstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:26:37.575Z
paper_authors._id: 63d209ae8eaa4831006b3b0d
paper_authors.avatarUrl: /avatars/51cc0f6dfab2233bddd73bfcd0c8aee3.svg
paper_authors.fullname: Jascha Sohl-Dickstein
paper_authors.type: user
paper_authors.user: jaschasd
paper_title: Frontier Language Models are not Robust to Adversarial Arithmetic, or
    "What do I need to say so you agree 2+2=5?
paper_summary: We introduce and study the problem of adversarial arithmetic, which provides
  a simple yet challenging testbed for language model alignment. This problem is
  comprised of arithmetic questions posed in natural language, with an arbitrary
  adversarial string inserted before the question is complete. Even in the simple
  setting of 1-digit addition problems, it is easy to find adversarial prompts
  that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and
  even to steer models to a particular wrong answer. We additionally provide a
  simple algorithm for finding successful attacks by querying those same models,
  which we name "prompt inversion rejection sampling" (PIRS). We finally show
  that models can be partially hardened against these attacks via reinforcement
  learning and via agentic constitutional loops. However, we were not able to
  make a language model fully robust against adversarial arithmetic attacks.
paper_ai_keywords: adversarial arithmetic language model alignment arithmetic questions natural language adversarial string 1-digit addition problems adversarial prompts PaLM2 GPT4 Claude2 prompt inversion rejection sampling (PIRS) reinforcement learning agentic constitutional loops