title: Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated
    Text
summary: Detecting text generated by modern large language models is thought to be
  hard, as both LLMs and humans can exhibit a wide range of complex behaviors.
  However, we find that a score based on contrasting two closely related language
  models is highly accurate at separating human-generated and machine-generated
  text. Based on this mechanism, we propose a novel LLM detector that only
  requires simple calculations using a pair of pre-trained LLMs. The method,
  called Binoculars, achieves state-of-the-art accuracy without any training
  data. It is capable of spotting machine text from a range of modern LLMs
  without any model-specific modifications. We comprehensively evaluate
  Binoculars on a number of text sources and in varied situations. Over a wide
  range of document types, Binoculars detects over 90% of generated samples from
  ChatGPT (and other LLMs) at a false positive rate of 0.01%, despite not being
  trained on any ChatGPT data.
paper_authors._id: 65af32e41edab235a1f3a9e0
paper_authors.name: Abhimanyu Hans
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:49:25.517Z
paper_authors._id: 6489c06c08b6a836a6dbd6aa
paper_authors.avatarUrl: /avatars/6ed1928992fc9d24bad0f12ebd2d84fe.svg
paper_authors.fullname: Abhimanyu Hans
paper_authors.type: user
paper_authors.user: ahans1
paper_authors._id: 65af32e41edab235a1f3a9e1
paper_authors.name: Avi Schwarzschild
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:49:32.749Z
paper_authors._id: 6570917c0ea91e592aff0b8c
paper_authors.avatarUrl: /avatars/529e9713e6ac835e11599ea7070a9603.svg
paper_authors.fullname: Avi Schwarzschild
paper_authors.type: user
paper_authors.user: schwarzschild
paper_authors._id: 65af32e41edab235a1f3a9e2
paper_authors.name: Valeriia Cherepanova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:49:39.483Z
paper_authors._id: 64c2ae06a2f9706201409fef
paper_authors.avatarUrl: /avatars/fb051d567411b88c7c0651051948e1d8.svg
paper_authors.fullname: Valeriia Cherepanova
paper_authors.type: user
paper_authors.user: vcherepanova
paper_authors._id: 65af32e41edab235a1f3a9e3
paper_authors.name: Hamid Kazemi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-15T16:42:46.467Z
paper_authors._id: 64f1f9612820a6f1b9e19edd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f1f9612820a6f1b9e19edd/hSmhLsczKR1t5jxHwEaE_.jpeg
paper_authors.fullname: Hamid Kazemi
paper_authors.type: user
paper_authors.user: seyedhamidreza
paper_authors._id: 65af32e41edab235a1f3a9e4
paper_authors.name: Aniruddha Saha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:49:49.385Z
paper_authors._id: 64178c29fff753e7c156a5da
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679264854378-64178c29fff753e7c156a5da.jpeg
paper_authors.fullname: Aniruddha Saha
paper_authors.type: user
paper_authors.user: ani0075saha
paper_authors._id: 65af32e41edab235a1f3a9e5
paper_authors.name: Micah Goldblum
paper_authors._id: 65af32e41edab235a1f3a9e6
paper_authors.name: Jonas Geiping
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:49:59.164Z
paper_authors._id: 63d86dbf3130cadcaf8bdd11
paper_authors.avatarUrl: /avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
paper_authors.fullname: Jonas Geiping
paper_authors.type: user
paper_authors.user: JonasGeiping
paper_authors._id: 65af32e41edab235a1f3a9e7
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:50:05.842Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_title: Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated
    Text
paper_summary: Detecting text generated by modern large language models is thought to be
  hard, as both LLMs and humans can exhibit a wide range of complex behaviors.
  However, we find that a score based on contrasting two closely related language
  models is highly accurate at separating human-generated and machine-generated
  text. Based on this mechanism, we propose a novel LLM detector that only
  requires simple calculations using a pair of pre-trained LLMs. The method,
  called Binoculars, achieves state-of-the-art accuracy without any training
  data. It is capable of spotting machine text from a range of modern LLMs
  without any model-specific modifications. We comprehensively evaluate
  Binoculars on a number of text sources and in varied situations. Over a wide
  range of document types, Binoculars detects over 90% of generated samples from
  ChatGPT (and other LLMs) at a false positive rate of 0.01%, despite not being
  trained on any ChatGPT data.
paper_ai_keywords: large language models (LLMs) contrasting pre-trained LLMs Binoculars state-of-the-art accuracy document types ChatGPT