title: Bring Your Own Data! Self-Supervised Evaluation for Large Language
    Models
summary: With the rise of Large Language Models (LLMs) and their ubiquitous deployment
  in diverse domains, measuring language model behavior on realistic data is
  imperative. For example, a company deploying a client-facing chatbot must
  ensure that the model will not respond to client requests with profanity.
  Current evaluations approach this problem using small, domain-specific datasets
  with human-curated labels. These evaluation sets are often sampled from a
  narrow and simplified distribution, and data sources can unknowingly be leaked
  into the training set which can lead to misleading evaluations. To bypass these
  drawbacks, we propose a framework for self-supervised evaluation of LLMs by
  analyzing their sensitivity or invariance to transformations on the input text.
  Self-supervised evaluation can directly monitor LLM behavior on datasets
  collected in the wild or streamed during live model deployment. We demonstrate
  self-supervised evaluation strategies for measuring closed-book knowledge,
  toxicity, and long-range context dependence, in addition to sensitivity to
  grammatical structure and tokenization errors. When comparisons to similar
  human-labeled benchmarks are available, we find strong correlations between
  self-supervised and human-supervised evaluations. The self-supervised paradigm
  complements current evaluation strategies that rely on labeled data.
paper_authors._id: 649918066117099581f69929
paper_authors.name: Neel Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T09:02:06.799Z
paper_authors._id: 63e2b1ec282ee5f9624cfbcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e2b1ec282ee5f9624cfbcb/4SVTp93cvRevacoJgiXzS.jpeg
paper_authors.fullname: Neel Jain
paper_authors.type: user
paper_authors.user: nsjain
paper_authors._id: 649918066117099581f6992a
paper_authors.name: Khalid Saifullah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T09:02:31.696Z
paper_authors._id: 603406de3d343696d96ef2a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/603406de3d343696d96ef2a6/FS3xGIC9G_JpyQ806VAyc.jpeg
paper_authors.fullname: Khalid Saifullah
paper_authors.type: user
paper_authors.user: khalidsaifullaah
paper_authors._id: 649918066117099581f6992b
paper_authors.name: Yuxin Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T09:02:52.790Z
paper_authors._id: 62cda27d248f9e6bc20c4261
paper_authors.avatarUrl: /avatars/907b9464071ff7ed1ff064331595f12f.svg
paper_authors.fullname: Yuxin Wen
paper_authors.type: user
paper_authors.user: ywen
paper_authors._id: 649918066117099581f6992c
paper_authors.name: John Kirchenbauer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T09:03:18.094Z
paper_authors._id: 63d98af1897746d6496177df
paper_authors.avatarUrl: /avatars/c5d0031c796a3c11bcb0d01b959168dc.svg
paper_authors.fullname: John Kirchenbauer
paper_authors.type: user
paper_authors.user: jwkirchenbauer
paper_authors._id: 649918066117099581f6992d
paper_authors.name: Manli Shu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T09:03:47.534Z
paper_authors._id: 63bdcb129a15a3e94198e7d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
paper_authors.fullname: Manli Shu
paper_authors.type: user
paper_authors.user: Manli
paper_authors._id: 649918066117099581f6992e
paper_authors.name: Aniruddha Saha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T09:04:06.225Z
paper_authors._id: 64178c29fff753e7c156a5da
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679264854378-64178c29fff753e7c156a5da.jpeg
paper_authors.fullname: Aniruddha Saha
paper_authors.type: user
paper_authors.user: ani0075saha
paper_authors._id: 649918066117099581f6992f
paper_authors.name: Micah Goldblum
paper_authors._id: 649918066117099581f69930
paper_authors.name: Jonas Geiping
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T09:05:07.056Z
paper_authors._id: 63d86dbf3130cadcaf8bdd11
paper_authors.avatarUrl: /avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
paper_authors.fullname: Jonas Geiping
paper_authors.type: user
paper_authors.user: JonasGeiping
paper_authors._id: 649918066117099581f69931
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T09:05:22.924Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_title: Bring Your Own Data! Self-Supervised Evaluation for Large Language
    Models
paper_summary: With the rise of Large Language Models (LLMs) and their ubiquitous deployment
  in diverse domains, measuring language model behavior on realistic data is
  imperative. For example, a company deploying a client-facing chatbot must
  ensure that the model will not respond to client requests with profanity.
  Current evaluations approach this problem using small, domain-specific datasets
  with human-curated labels. These evaluation sets are often sampled from a
  narrow and simplified distribution, and data sources can unknowingly be leaked
  into the training set which can lead to misleading evaluations. To bypass these
  drawbacks, we propose a framework for self-supervised evaluation of LLMs by
  analyzing their sensitivity or invariance to transformations on the input text.
  Self-supervised evaluation can directly monitor LLM behavior on datasets
  collected in the wild or streamed during live model deployment. We demonstrate
  self-supervised evaluation strategies for measuring closed-book knowledge,
  toxicity, and long-range context dependence, in addition to sensitivity to
  grammatical structure and tokenization errors. When comparisons to similar
  human-labeled benchmarks are available, we find strong correlations between
  self-supervised and human-supervised evaluations. The self-supervised paradigm
  complements current evaluation strategies that rely on labeled data.
paper_ai_summary: A self-supervised evaluation framework quantifies large language model behavior without human-curated labels by analyzing text transformations.
paper_ai_keywords: Large Language Models LLMs self-supervised evaluation closed-book knowledge toxicity long-range context dependence grammatical structure tokenization errors human-labeled benchmarks