title: TLDR: Token-Level Detective Reward Model for Large Vision Language
    Models
summary: Although reward models have been successful in improving multimodal large
  language models, the reward models themselves remain brutal and contain minimal
  information. Notably, existing reward models only mimic human annotations by
  assigning only one binary feedback to any text, no matter how long the text is.
  In the realm of multimodal language models, where models are required to
  process both images and texts, a naive reward model may learn implicit biases
  toward texts and become less grounded in images. In this paper, we propose a
  Token-Level Detective Reward Model
  (TLDR) to provide fine-grained annotations to each text token. We
  first introduce a perturbation-based method to generate synthetic hard
  negatives and their token-level labels to train TLDR models. Then we show the
  rich usefulness of TLDR models both in assisting off-the-shelf models to
  self-correct their generations, and in serving as a hallucination evaluation
  tool. Finally, we show that TLDR models can significantly speed up human
  annotation by 3 times to acquire a broader range of high-quality vision
  language data.
paper_authors._id: 6704b02e6282fcabb75d684e
paper_authors.name: Deqing Fu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-08T04:10:22.462Z
paper_authors._id: 63c8454e46421a2efe82709d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/3BcSk4KOwAgWHEPVtsAV3.png
paper_authors.fullname: Deqing Fu
paper_authors.type: user
paper_authors.user: deqing
paper_authors._id: 6704b02e6282fcabb75d684f
paper_authors.name: Tong Xiao
paper_authors._id: 6704b02e6282fcabb75d6850
paper_authors.name: Rui Wang
paper_authors._id: 6704b02e6282fcabb75d6851
paper_authors.name: Wang Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:32.657Z
paper_authors._id: 6389568eec1f539adc0e0337
paper_authors.avatarUrl: /avatars/6ee5c9bea9989e4618ae8d334bd889a9.svg
paper_authors.fullname: Wang Bill Zhu
paper_authors.type: user
paper_authors.user: Bill1235813
paper_authors._id: 6704b02e6282fcabb75d6852
paper_authors.name: Pengchuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:40:10.427Z
paper_authors._id: 640056f5261cfa61f39b4c35
paper_authors.avatarUrl: /avatars/25aa5dee42caa69327d158a669f6bf3a.svg
paper_authors.fullname: Pengchuan Zhang
paper_authors.type: user
paper_authors.user: pzzhang
paper_authors._id: 6704b02e6282fcabb75d6853
paper_authors.name: Guan Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:40:04.655Z
paper_authors._id: 6573a458d7f487de5f9bb44a
paper_authors.avatarUrl: /avatars/4aa5626af43855ad770186f0d4191a01.svg
paper_authors.fullname: Guan Pang
paper_authors.type: user
paper_authors.user: tbpangolin
paper_authors._id: 6704b02e6282fcabb75d6854
paper_authors.name: Robin Jia
paper_authors._id: 6704b02e6282fcabb75d6855
paper_authors.name: Lawrence Chen
paper_title: TLDR: Token-Level Detective Reward Model for Large Vision Language
    Models
paper_summary: Although reward models have been successful in improving multimodal large
  language models, the reward models themselves remain brutal and contain minimal
  information. Notably, existing reward models only mimic human annotations by
  assigning only one binary feedback to any text, no matter how long the text is.
  In the realm of multimodal language models, where models are required to
  process both images and texts, a naive reward model may learn implicit biases
  toward texts and become less grounded in images. In this paper, we propose a
  Token-Level Detective Reward Model
  (TLDR) to provide fine-grained annotations to each text token. We
  first introduce a perturbation-based method to generate synthetic hard
  negatives and their token-level labels to train TLDR models. Then we show the
  rich usefulness of TLDR models both in assisting off-the-shelf models to
  self-correct their generations, and in serving as a hallucination evaluation
  tool. Finally, we show that TLDR models can significantly speed up human
  annotation by 3 times to acquire a broader range of high-quality vision
  language data.
paper_ai_summary: A token-level detective reward model (TLDR) provides detailed feedback for text tokens, assisting in self-correction and hallucination evaluation, and accelerating human annotation.
paper_ai_keywords: reward models multimodal language models token-level perturbation-based synthetic hard negatives hallucination evaluation