Mitigating Object Hallucinations via Sentence-Level Early Intervention
Multimodal large language models (MLLMs) have revolutionized cross-modal
understanding but continue to struggle with hallucinations - fabricated content
contradicting visual inputs. Existing hallucination mitigation methods either
incur prohibitive computational costs or introduce distribution mismatches
between training data and model outputs. We identify a critical insight:
hallucinations predominantly emerge at the early stages of text generation and
propagate through subsequent outputs. To address this, we propose **SENTINEL**
(**S**entence-level **E**arly i**N**tervention **T**hrough **IN**-domain
pr**E**ference **L**earning), a framework that eliminates dependency on human
annotations. Specifically, we first bootstrap high-quality in-domain preference
pairs by iteratively sampling model outputs, validating object existence
through cross-checking with two open-vocabulary detectors, and classifying
sentences into hallucinated/non-hallucinated categories. Subsequently, we use
context-coherent positive samples and hallucinated negative samples to build
context-aware preference data iteratively. Finally, we train models using a
context-aware preference loss (C-DPO) that emphasizes discriminative learning
at the sentence level where hallucinations initially manifest. Experimental
results show that SENTINEL can reduce hallucinations by over 90\% compared to
the original model and outperforms the previous state-of-the-art method on both
hallucination benchmarks and general capabilities benchmarks, demonstrating its
superiority and generalization ability. The models, datasets, and code are
available at https://github.com/pspdada/SENTINEL.
687e0ea52e8db0930be6f1af
Shangpin Peng
claimed_verified
2025-07-21T14:21:53.355Z
66b6df9d512dac2ac07cf0c9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/z0w4X-CdeEoFnwvgrsO9Z.jpeg
Peng Shangpin
user
psp-dada
687e0ea52e8db0930be6f1b0
Senqiao Yang
admin_assigned
2025-07-21T16:04:23.909Z
6527b7280ae663e384eb8499
https://cdn-avatars.huggingface.co/v1/production/uploads/6527b7280ae663e384eb8499/73yF3eu2cUx7jVZrhXnXx.jpeg
Senqiao Yang
user
Senqiao
687e0ea52e8db0930be6f1b1
Li Jiang
687e0ea52e8db0930be6f1b2
Zhuotao Tian
Mitigating Object Hallucinations via Sentence-Level Early Intervention
Multimodal large language models (MLLMs) have revolutionized cross-modal
understanding but continue to struggle with hallucinations - fabricated content
contradicting visual inputs. Existing hallucination mitigation methods either
incur prohibitive computational costs or introduce distribution mismatches
between training data and model outputs. We identify a critical insight:
hallucinations predominantly emerge at the early stages of text generation and
propagate through subsequent outputs. To address this, we propose **SENTINEL**
(**S**entence-level **E**arly i**N**tervention **T**hrough **IN**-domain
pr**E**ference **L**earning), a framework that eliminates dependency on human
annotations. Specifically, we first bootstrap high-quality in-domain preference
pairs by iteratively sampling model outputs, validating object existence
through cross-checking with two open-vocabulary detectors, and classifying
sentences into hallucinated/non-hallucinated categories. Subsequently, we use
context-coherent positive samples and hallucinated negative samples to build
context-aware preference data iteratively. Finally, we train models using a
context-aware preference loss (C-DPO) that emphasizes discriminative learning
at the sentence level where hallucinations initially manifest. Experimental
results show that SENTINEL can reduce hallucinations by over 90\% compared to
the original model and outperforms the previous state-of-the-art method on both
hallucination benchmarks and general capabilities benchmarks, demonstrating its
superiority and generalization ability. The models, datasets, and code are
available at https://github.com/pspdada/SENTINEL.
SENTINEL reduces hallucinations in multimodal large language models by iteratively generating and validating sentence-level outputs using cross-checking detectors and training with a context-aware preference loss.
multimodal large language models hallucinations SENTINEL in-domain preference learning open-vocabulary detectors context-aware preference data context-aware preference loss discriminative learning