title: Guiding Image Captioning Models Toward More Specific Captions
summary: Image captioning is conventionally formulated as the task of generating
  captions for images that match the distribution of reference image-caption
  pairs. However, reference captions in standard captioning datasets are short
  and may not uniquely identify the images they describe. These problems are
  further exacerbated when models are trained directly on image-alt text pairs
  collected from the internet. In this work, we show that it is possible to
  generate more specific captions with minimal changes to the training process.
  We implement classifier-free guidance for an autoregressive captioning model by
  fine-tuning it to estimate both conditional and unconditional distributions
  over captions. The guidance scale applied at decoding controls a trade-off
  between maximizing p(caption|image) and
  p(image|caption). Compared to standard greedy decoding,
  decoding with a guidance scale of 2 substantially improves reference-free
  metrics such as CLIPScore (0.808 vs. 0.775) and captiontoimage retrieval
  performance in the CLIP embedding space (recall@1 44.6% vs. 26.5%), but worsens
  standard reference-based captioning metrics (e.g., CIDEr 78.6 vs 126.1). We
  further explore the use of language models to guide the decoding process,
  obtaining small improvements over the Pareto frontier of reference-free vs.
  reference-based captioning metrics that arises from classifier-free guidance,
  and substantially improving the quality of captions generated from a model
  trained only on minimally curated web data.
paper_authors._id: 64c86d027b4d0d947cd5292e
paper_authors.name: Simon Kornblith
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T09:26:26.786Z
paper_authors._id: 6443ffc2af034cdfd69c5c00
paper_authors.avatarUrl: /avatars/0c12cf9baaf3b24a02ffb42238cdc6df.svg
paper_authors.fullname: Simon Kornblith
paper_authors.type: user
paper_authors.user: simonsterk
paper_authors._id: 64c86d027b4d0d947cd5292f
paper_authors.name: Lala Li
paper_authors._id: 64c86d027b4d0d947cd52930
paper_authors.name: Zirui Wang
paper_authors._id: 64c86d027b4d0d947cd52931
paper_authors.name: Thao Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T07:44:25.132Z
paper_authors._id: 64ba4923921dbcbc96006fb8
paper_authors.avatarUrl: /avatars/4eab3991e8502f72ddf8ca4825c2ad65.svg
paper_authors.fullname: Thao Nguyen
paper_authors.type: user
paper_authors.user: thaottn
paper_title: Guiding Image Captioning Models Toward More Specific Captions
paper_summary: Image captioning is conventionally formulated as the task of generating
  captions for images that match the distribution of reference image-caption
  pairs. However, reference captions in standard captioning datasets are short
  and may not uniquely identify the images they describe. These problems are
  further exacerbated when models are trained directly on image-alt text pairs
  collected from the internet. In this work, we show that it is possible to
  generate more specific captions with minimal changes to the training process.
  We implement classifier-free guidance for an autoregressive captioning model by
  fine-tuning it to estimate both conditional and unconditional distributions
  over captions. The guidance scale applied at decoding controls a trade-off
  between maximizing p(caption|image) and
  p(image|caption). Compared to standard greedy decoding,
  decoding with a guidance scale of 2 substantially improves reference-free
  metrics such as CLIPScore (0.808 vs. 0.775) and captiontoimage retrieval
  performance in the CLIP embedding space (recall@1 44.6% vs. 26.5%), but worsens
  standard reference-based captioning metrics (e.g., CIDEr 78.6 vs 126.1). We
  further explore the use of language models to guide the decoding process,
  obtaining small improvements over the Pareto frontier of reference-free vs.
  reference-based captioning metrics that arises from classifier-free guidance,
  and substantially improving the quality of captions generated from a model
  trained only on minimally curated web data.
paper_ai_keywords: classifier-free guidance autoregressive captioning model conditional distribution unconditional distribution CLIPScore caption$\to$image retrieval CLIP embedding space recall@1 CIDEr