Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of
  Language Models
We investigate the internal behavior of Transformer-based Large Language
Models (LLMs) when they generate factually incorrect text. We propose modeling
factual queries as Constraint Satisfaction Problems and use this framework to
investigate how the model interacts internally with factual constraints.
Specifically, we discover a strong positive relation between the model's
attention to constraint tokens and the factual accuracy of its responses. In
our curated suite of 11 datasets with over 40,000 prompts, we study the task of
predicting factual errors with the Llama-2 family across all scales (7B, 13B,
70B). We propose SAT Probe, a method probing self-attention patterns, that can
predict constraint satisfaction and factual errors, and allows early error
identification. The approach and findings demonstrate how using the mechanistic
understanding of factuality in LLMs can enhance reliability.
65139c88bd21724fd44f1740
Mert Yuksekgonul
extracted_confirmed
2024-06-12T03:35:58.562Z
61eb17ac8ae3ccab3b3da97b
/avatars/9c91c33a0744eae1454e242d51295fbc.svg
Mert
user
merty
65139c88bd21724fd44f1741
Varun Chandrasekaran
65139c88bd21724fd44f1742
Erik Jones
admin_assigned
2023-09-27T10:19:35.272Z
63518b69d95a8aa0c2247488
/avatars/8ab0cb1e4e5d66c0018e22df035a6a9e.svg
Erik Jones
user
erikjones313
65139c88bd21724fd44f1743
Suriya Gunasekar
admin_assigned
2023-09-27T10:19:42.961Z
63e1b4f77fbb6ae4d4f36aa4
/avatars/7febc332d393e0bb94255104527ffe2a.svg
Gunasekar
user
suriyagunasekar
65139c88bd21724fd44f1744
Ranjita Naik
admin_assigned
2023-09-27T10:19:55.434Z
63d8512207cd1aa3c4a1a835
/avatars/cd5056c920a6b18eb93e2e9d0c5dc7c4.svg
Ranjita Naik
user
ranjita
65139c88bd21724fd44f1745
Hamid Palangi
admin_assigned
2023-09-27T10:20:03.661Z
6189ddd0992df2640e3e7d40
/avatars/925f2308fc412ae352b57a1b71815028.svg
Hamid Palangi
user
hamidpalangi
65139c88bd21724fd44f1746
Ece Kamar
65139c88bd21724fd44f1747
Besmira Nushi
claimed_verified
2023-10-02T07:00:24.772Z
638ee2ed848625f9fad11034
https://cdn-avatars.huggingface.co/v1/production/uploads/1670309301345-638ee2ed848625f9fad11034.jpeg
Besmira Nushi
user
nushib
Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of
  Language Models
We investigate the internal behavior of Transformer-based Large Language
Models (LLMs) when they generate factually incorrect text. We propose modeling
factual queries as Constraint Satisfaction Problems and use this framework to
investigate how the model interacts internally with factual constraints.
Specifically, we discover a strong positive relation between the model's
attention to constraint tokens and the factual accuracy of its responses. In
our curated suite of 11 datasets with over 40,000 prompts, we study the task of
predicting factual errors with the Llama-2 family across all scales (7B, 13B,
70B). We propose SAT Probe, a method probing self-attention patterns, that can
predict constraint satisfaction and factual errors, and allows early error
identification. The approach and findings demonstrate how using the mechanistic
understanding of factuality in LLMs can enhance reliability.
Transformer-based LLMs show a positive correlation between attention to constraint tokens and factual accuracy, and SAT Probe can predict factual errors by analyzing self-attention patterns.
Transformer-based Large Language Models (LLMs) Constraint Satisfaction Problems factual accuracy self-attention patterns SAT Probe mechanistic understanding