I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token
Large Language Models are known to capture real-world knowledge, allowing
them to excel in many downstream tasks. Despite recent advances, these models
are still prone to what are commonly known as hallucinations, causing them to
emit unwanted and factually incorrect text. In this work, we propose a novel
calibration method that can be used to combat hallucinations. We add a special
[IDK] ("I don't know") token to the model's vocabulary and introduce an
objective function that shifts probability mass to the [IDK] token for
incorrect predictions. This approach allows the model to express uncertainty in
its output explicitly. We evaluate our proposed method across multiple model
architectures and factual downstream tasks. We find that models trained with
our method are able to express uncertainty in places where they would
previously make mistakes while suffering only a small loss of encoded
knowledge. We further perform extensive ablation studies of multiple variations
of our approach and provide a detailed analysis of the precision-recall
tradeoff of our method.
6759fcc41f49e61f776bb7ce
Roi Cohen
extracted_confirmed
2024-12-11T21:01:39.374Z
65b7afaeb4ec059b974e9189
/avatars/08059754f2d299b76ab4649e8d768253.svg
Roi Cohen
user
roicohen9
6759fcc41f49e61f776bb7cf
Konstantin Dobler
claimed_verified
2024-12-13T07:30:55.400Z
62a33fe73621790e22b26c87
https://cdn-avatars.huggingface.co/v1/production/uploads/62a33fe73621790e22b26c87/U5LnpPTlWcdshWFch2adD.jpeg
Konstantin Dobler
user
konstantindobler
6759fcc41f49e61f776bb7d0
Eden Biran
6759fcc41f49e61f776bb7d1
Gerard de Melo
I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token
Large Language Models are known to capture real-world knowledge, allowing
them to excel in many downstream tasks. Despite recent advances, these models
are still prone to what are commonly known as hallucinations, causing them to
emit unwanted and factually incorrect text. In this work, we propose a novel
calibration method that can be used to combat hallucinations. We add a special
[IDK] ("I don't know") token to the model's vocabulary and introduce an
objective function that shifts probability mass to the [IDK] token for
incorrect predictions. This approach allows the model to express uncertainty in
its output explicitly. We evaluate our proposed method across multiple model
architectures and factual downstream tasks. We find that models trained with
our method are able to express uncertainty in places where they would
previously make mistakes while suffering only a small loss of encoded
knowledge. We further perform extensive ablation studies of multiple variations
of our approach and provide a detailed analysis of the precision-recall
tradeoff of our method.
A novel calibration method using an [IDK] token reduces hallucinations in large language models while maintaining knowledge accuracy.
IDK token calibration method hallucinations probability mass uncertainty precision-recall tradeoff