title: Simple Token-Level Confidence Improves Caption Correctness
summary: The ability to judge whether a caption correctly describes an image is a
  critical part of vision-language understanding. However, state-of-the-art
  models often misinterpret the correctness of fine-grained details, leading to
  errors in outputs such as hallucinating objects in generated captions or poor
  compositional reasoning. In this work, we explore Token-Level Confidence, or
  TLC, as a simple yet surprisingly effective method to assess caption
  correctness. Specifically, we fine-tune a vision-language model on image
  captioning, input an image and proposed caption to the model, and aggregate
  either algebraic or learned token confidences over words or sequences to
  estimate image-caption consistency. Compared to sequence-level scores from
  pretrained models, TLC with algebraic confidence measures achieves a relative
  improvement in accuracy by 10% on verb understanding in SVO-Probes and
  outperforms prior state-of-the-art in image and group scores for compositional
  reasoning in Winoground by a relative 37% and 9%, respectively. When training
  data are available, a learned confidence estimator provides further improved
  performance, reducing object hallucination rates in MS COCO Captions by a
  relative 30% over the original model and setting a new state-of-the-art.
paper_authors._id: 645dbd1ca19f3e64bbf31dc5
paper_authors.name: Suzanne Petryk
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T09:55:49.564Z
paper_authors._id: 6361312b3c7147fae7dfecd2
paper_authors.avatarUrl: /avatars/df7d1596191b98da2d2a01d5bbb880c3.svg
paper_authors.fullname: Suzie Petryk
paper_authors.type: user
paper_authors.user: spetryk
paper_authors._id: 645dbd1ca19f3e64bbf31dc6
paper_authors.name: Spencer Whitehead
paper_authors._id: 645dbd1ca19f3e64bbf31dc7
paper_authors.name: Joseph E. Gonzalez
paper_authors._id: 645dbd1ca19f3e64bbf31dc8
paper_authors.name: Trevor Darrell
paper_authors._id: 645dbd1ca19f3e64bbf31dc9
paper_authors.name: Anna Rohrbach
paper_authors._id: 645dbd1ca19f3e64bbf31dca
paper_authors.name: Marcus Rohrbach
paper_title: Simple Token-Level Confidence Improves Caption Correctness
paper_summary: The ability to judge whether a caption correctly describes an image is a
  critical part of vision-language understanding. However, state-of-the-art
  models often misinterpret the correctness of fine-grained details, leading to
  errors in outputs such as hallucinating objects in generated captions or poor
  compositional reasoning. In this work, we explore Token-Level Confidence, or
  TLC, as a simple yet surprisingly effective method to assess caption
  correctness. Specifically, we fine-tune a vision-language model on image
  captioning, input an image and proposed caption to the model, and aggregate
  either algebraic or learned token confidences over words or sequences to
  estimate image-caption consistency. Compared to sequence-level scores from
  pretrained models, TLC with algebraic confidence measures achieves a relative
  improvement in accuracy by 10% on verb understanding in SVO-Probes and
  outperforms prior state-of-the-art in image and group scores for compositional
  reasoning in Winoground by a relative 37% and 9%, respectively. When training
  data are available, a learned confidence estimator provides further improved
  performance, reducing object hallucination rates in MS COCO Captions by a
  relative 30% over the original model and setting a new state-of-the-art.
paper_ai_summary: Token-Level Confidence (TLC) improves caption correctness by assessing fine-grained details, reducing errors like object hallucination and enhancing compositional reasoning.
paper_ai_keywords: Token-Level Confidence TLC fine-grained details hallucinating objects compositional reasoning sequence-level scores pretrained models verb understanding SVO-Probes Winoground MS COCO Captions