TARS: MinMax Token-Adaptive Preference Strategy for Hallucination
  Reduction in MLLMs
Multimodal large language models (MLLMs) enable vision-language reasoning,
yet often generate plausible outputs that are factually incorrect or visually
ungrounded, thereby compromising their reliability. Direct preference
optimization (DPO) is a common strategy for correcting hallucinations by
aligning model outputs with human preferences. Existing DPO strategies
typically treat hallucination-related preferences as fixed targets, relying on
static supervision signals during training. This approach tends to overfit to
superficial linguistic cues in preference data, leading to distributional
rigidity and spurious correlations that impair grounding in causally relevant
visual information. To overcome this limitation, we propose TARS, a
token-adaptive preference strategy that reformulates DPO as a min-max
optimization problem. TARS maximizes token-level distributional shifts under
semantic constraints to simulate alignment uncertainty, and simultaneously
minimizes the expected preference loss under these controlled perturbations.
This joint objective preserves causal grounding while mitigating overfitting to
preference patterns, thereby reducing hallucinations in multimodal reasoning.
We evaluate TARS on multiple hallucination benchmarks and find consistently
strong performance. Using only 4.8k preference samples and no expert feedback,
TARS reduces hallucination rates from 26.4% to 13.2% and decreases cognition
value from 2.5 to 0.4. It outperforms standard DPO and matches GPT-4o on
several key metrics.
688c690b8c434640078cc4ad
Kejia Zhang
688c690b8c434640078cc4ae
Keda Tao
688c690b8c434640078cc4af
Zhiming Luo
688c690b8c434640078cc4b0
Chang Liu
688c690b8c434640078cc4b1
Jiasheng Tang
688c690b8c434640078cc4b2
Huan Wang
TARS: MinMax Token-Adaptive Preference Strategy for Hallucination
  Reduction in MLLMs
Multimodal large language models (MLLMs) enable vision-language reasoning,
yet often generate plausible outputs that are factually incorrect or visually
ungrounded, thereby compromising their reliability. Direct preference
optimization (DPO) is a common strategy for correcting hallucinations by
aligning model outputs with human preferences. Existing DPO strategies
typically treat hallucination-related preferences as fixed targets, relying on
static supervision signals during training. This approach tends to overfit to
superficial linguistic cues in preference data, leading to distributional
rigidity and spurious correlations that impair grounding in causally relevant
visual information. To overcome this limitation, we propose TARS, a
token-adaptive preference strategy that reformulates DPO as a min-max
optimization problem. TARS maximizes token-level distributional shifts under
semantic constraints to simulate alignment uncertainty, and simultaneously
minimizes the expected preference loss under these controlled perturbations.
This joint objective preserves causal grounding while mitigating overfitting to
preference patterns, thereby reducing hallucinations in multimodal reasoning.
We evaluate TARS on multiple hallucination benchmarks and find consistently
strong performance. Using only 4.8k preference samples and no expert feedback,
TARS reduces hallucination rates from 26.4% to 13.2% and decreases cognition
value from 2.5 to 0.4. It outperforms standard DPO and matches GPT-4o on
several key metrics.
TARS, a token-adaptive preference strategy, improves multimodal large language models by reducing hallucinations through min-max optimization under semantic constraints.
multimodal large language models vision-language reasoning direct preference optimization hallucinations token-adaptive preference strategy min-max optimization distributional shifts semantic constraints preference loss causal grounding overfitting hallucination benchmarks cognition value GPT-4o