Bring Your Own Data! Self-Supervised Evaluation for Large Language
  Models
With the rise of Large Language Models (LLMs) and their ubiquitous deployment
in diverse domains, measuring language model behavior on realistic data is
imperative. For example, a company deploying a client-facing chatbot must
ensure that the model will not respond to client requests with profanity.
Current evaluations approach this problem using small, domain-specific datasets
with human-curated labels. These evaluation sets are often sampled from a
narrow and simplified distribution, and data sources can unknowingly be leaked
into the training set which can lead to misleading evaluations. To bypass these
drawbacks, we propose a framework for self-supervised evaluation of LLMs by
analyzing their sensitivity or invariance to transformations on the input text.
Self-supervised evaluation can directly monitor LLM behavior on datasets
collected in the wild or streamed during live model deployment. We demonstrate
self-supervised evaluation strategies for measuring closed-book knowledge,
toxicity, and long-range context dependence, in addition to sensitivity to
grammatical structure and tokenization errors. When comparisons to similar
human-labeled benchmarks are available, we find strong correlations between
self-supervised and human-supervised evaluations. The self-supervised paradigm
complements current evaluation strategies that rely on labeled data.
649918066117099581f69929
Neel Jain
admin_assigned
2023-06-26T09:02:06.799Z
63e2b1ec282ee5f9624cfbcb
https://cdn-avatars.huggingface.co/v1/production/uploads/63e2b1ec282ee5f9624cfbcb/4SVTp93cvRevacoJgiXzS.jpeg
Neel Jain
user
nsjain
649918066117099581f6992a
Khalid Saifullah
admin_assigned
2023-06-26T09:02:31.696Z
603406de3d343696d96ef2a6
https://cdn-avatars.huggingface.co/v1/production/uploads/603406de3d343696d96ef2a6/FS3xGIC9G_JpyQ806VAyc.jpeg
Khalid Saifullah
user
khalidsaifullaah
649918066117099581f6992b
Yuxin Wen
admin_assigned
2023-06-26T09:02:52.790Z
62cda27d248f9e6bc20c4261
/avatars/907b9464071ff7ed1ff064331595f12f.svg
Yuxin Wen
user
ywen
649918066117099581f6992c
John Kirchenbauer
admin_assigned
2023-06-26T09:03:18.094Z
63d98af1897746d6496177df
/avatars/c5d0031c796a3c11bcb0d01b959168dc.svg
John Kirchenbauer
user
jwkirchenbauer
649918066117099581f6992d
Manli Shu
admin_assigned
2023-06-26T09:03:47.534Z
63bdcb129a15a3e94198e7d3
https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
Manli Shu
user
Manli
649918066117099581f6992e
Aniruddha Saha
admin_assigned
2023-06-26T09:04:06.225Z
64178c29fff753e7c156a5da
https://cdn-avatars.huggingface.co/v1/production/uploads/1679264854378-64178c29fff753e7c156a5da.jpeg
Aniruddha Saha
user
ani0075saha
649918066117099581f6992f
Micah Goldblum
649918066117099581f69930
Jonas Geiping
admin_assigned
2023-06-26T09:05:07.056Z
63d86dbf3130cadcaf8bdd11
/avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
Jonas Geiping
user
JonasGeiping
649918066117099581f69931
Tom Goldstein
admin_assigned
2023-06-26T09:05:22.924Z
6381ca7d65dc156aba0b933d
/avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
Tom Goldstein
user
tomgoldstein
Bring Your Own Data! Self-Supervised Evaluation for Large Language
  Models
With the rise of Large Language Models (LLMs) and their ubiquitous deployment
in diverse domains, measuring language model behavior on realistic data is
imperative. For example, a company deploying a client-facing chatbot must
ensure that the model will not respond to client requests with profanity.
Current evaluations approach this problem using small, domain-specific datasets
with human-curated labels. These evaluation sets are often sampled from a
narrow and simplified distribution, and data sources can unknowingly be leaked
into the training set which can lead to misleading evaluations. To bypass these
drawbacks, we propose a framework for self-supervised evaluation of LLMs by
analyzing their sensitivity or invariance to transformations on the input text.
Self-supervised evaluation can directly monitor LLM behavior on datasets
collected in the wild or streamed during live model deployment. We demonstrate
self-supervised evaluation strategies for measuring closed-book knowledge,
toxicity, and long-range context dependence, in addition to sensitivity to
grammatical structure and tokenization errors. When comparisons to similar
human-labeled benchmarks are available, we find strong correlations between
self-supervised and human-supervised evaluations. The self-supervised paradigm
complements current evaluation strategies that rely on labeled data.
A self-supervised evaluation framework quantifies large language model behavior without human-curated labels by analyzing text transformations.
Large Language Models LLMs self-supervised evaluation closed-book knowledge toxicity long-range context dependence grammatical structure tokenization errors human-labeled benchmarks