title: Understanding Retrieval Robustness for Retrieval-Augmented Image
    Captioning
summary: Recent advances in retrieval-augmented models for image captioning highlight
  the benefit of retrieving related captions for efficient, lightweight models
  with strong domain-transfer capabilities. While these models demonstrate the
  success of retrieval augmentation, retrieval models are still far from perfect
  in practice: the retrieved information can sometimes mislead the model,
  resulting in incorrect generation and worse performance. In this paper, we
  analyze the robustness of a retrieval-augmented captioning model SmallCap. Our
  analysis shows that the model is sensitive to tokens that appear in the
  majority of the retrieved captions, and the input attribution shows that those
  tokens are likely copied into the generated output. Given these findings, we
  propose to train the model by sampling retrieved captions from more diverse
  sets. This decreases the chance that the model learns to copy majority tokens,
  and improves both in-domain and cross-domain performance.
paper_authors._id: 668f02f92eb49d45ac6a4f2a
paper_authors.name: Wenyan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T09:01:01.526Z
paper_authors._id: 619b506f70d03780cbec5806
paper_authors.avatarUrl: /avatars/ea2b0b8f0a3eb16d53ef40da9981c397.svg
paper_authors.fullname: wenyan li
paper_authors.type: user
paper_authors.user: lyan62
paper_authors._id: 668f02f92eb49d45ac6a4f2b
paper_authors.name: Jiaang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T08:26:59.087Z
paper_authors._id: 6686658221f096c7fb4e0b24
paper_authors.avatarUrl: /avatars/581f5d56b393b95c74f38a06f6bb4f69.svg
paper_authors.fullname: Jiaang Li
paper_authors.type: user
paper_authors.user: jaagli
paper_authors._id: 668f02f92eb49d45ac6a4f2c
paper_authors.name: Rita Ramos
paper_authors._id: 668f02f92eb49d45ac6a4f2d
paper_authors.name: Raphael Tang
paper_authors._id: 668f02f92eb49d45ac6a4f2e
paper_authors.name: Desmond Elliott
paper_title: Understanding Retrieval Robustness for Retrieval-Augmented Image
    Captioning
paper_summary: Recent advances in retrieval-augmented models for image captioning highlight
  the benefit of retrieving related captions for efficient, lightweight models
  with strong domain-transfer capabilities. While these models demonstrate the
  success of retrieval augmentation, retrieval models are still far from perfect
  in practice: the retrieved information can sometimes mislead the model,
  resulting in incorrect generation and worse performance. In this paper, we
  analyze the robustness of a retrieval-augmented captioning model SmallCap. Our
  analysis shows that the model is sensitive to tokens that appear in the
  majority of the retrieved captions, and the input attribution shows that those
  tokens are likely copied into the generated output. Given these findings, we
  propose to train the model by sampling retrieved captions from more diverse
  sets. This decreases the chance that the model learns to copy majority tokens,
  and improves both in-domain and cross-domain performance.
paper_ai_keywords: retrieval-augmented models image captioning retrieval models domain-transfer capabilities token attribution caption generation model sensitivity majority tokens diverse sets