title: Improving fine-grained understanding in image-text pre-training
summary: We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple
  method for pretraining more fine-grained multimodal representations from
  image-text pairs. Given that multiple image patches often correspond to single
  words, we propose to learn a grouping of image patches for every token in the
  caption. To achieve this, we use a sparse similarity metric between image
  patches and language tokens and compute for each token a language-grouped
  vision embedding as the weighted average of patches. The token and
  language-grouped vision embeddings are then contrasted through a fine-grained
  sequence-wise loss that only depends on individual samples and does not require
  other batch samples as negatives. This enables more detailed information to be
  learned in a computationally inexpensive manner. SPARC combines this
  fine-grained loss with a contrastive loss between global image and text
  embeddings to learn representations that simultaneously encode global and local
  information. We thoroughly evaluate our proposed method and show improved
  performance over competing approaches both on image-level tasks relying on
  coarse-grained information, e.g. classification, as well as region-level tasks
  relying on fine-grained information, e.g. retrieval, object detection, and
  segmentation. Moreover, SPARC improves model faithfulness and captioning in
  foundational vision-language models.
paper_authors._id: 65a9e02ae15948f4c6b25dc7
paper_authors.name: Ioana Bica
paper_authors._id: 65a9e02ae15948f4c6b25dc8
paper_authors.name: Anastasija Ilić
paper_authors._id: 65a9e02ae15948f4c6b25dc9
paper_authors.name: Matthias Bauer
paper_authors._id: 65a9e02ae15948f4c6b25dca
paper_authors.name: Goker Erdogan
paper_authors._id: 65a9e02ae15948f4c6b25dcb
paper_authors.name: Matko Bošnjak
paper_authors._id: 65a9e02ae15948f4c6b25dcc
paper_authors.name: Christos Kaplanis
paper_authors._id: 65a9e02ae15948f4c6b25dcd
paper_authors.name: Alexey A. Gritsenko
paper_authors._id: 65a9e02ae15948f4c6b25dce
paper_authors.name: Matthias Minderer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:27:42.915Z
paper_authors._id: 649ac6e57c36fc2dc6e6b0f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649ac6e57c36fc2dc6e6b0f4/Scd3LiYW5Y2qunDDHnPWw.jpeg
paper_authors.fullname: Matthias Minderer
paper_authors.type: user
paper_authors.user: mjlm
paper_authors._id: 65a9e02ae15948f4c6b25dcf
paper_authors.name: Charles Blundell
paper_authors._id: 65a9e02ae15948f4c6b25dd0
paper_authors.name: Razvan Pascanu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:27:54.801Z
paper_authors._id: 64b9310403124195cd9778ec
paper_authors.avatarUrl: /avatars/57c594d3d0f97d3010b15b6a0806451c.svg
paper_authors.fullname: Razvan Pascanu
paper_authors.type: user
paper_authors.user: razp
paper_authors._id: 65a9e02ae15948f4c6b25dd1
paper_authors.name: Jovana Mitrović
paper_title: Improving fine-grained understanding in image-text pre-training
paper_summary: We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple
  method for pretraining more fine-grained multimodal representations from
  image-text pairs. Given that multiple image patches often correspond to single
  words, we propose to learn a grouping of image patches for every token in the
  caption. To achieve this, we use a sparse similarity metric between image
  patches and language tokens and compute for each token a language-grouped
  vision embedding as the weighted average of patches. The token and
  language-grouped vision embeddings are then contrasted through a fine-grained
  sequence-wise loss that only depends on individual samples and does not require
  other batch samples as negatives. This enables more detailed information to be
  learned in a computationally inexpensive manner. SPARC combines this
  fine-grained loss with a contrastive loss between global image and text
  embeddings to learn representations that simultaneously encode global and local
  information. We thoroughly evaluate our proposed method and show improved
  performance over competing approaches both on image-level tasks relying on
  coarse-grained information, e.g. classification, as well as region-level tasks
  relying on fine-grained information, e.g. retrieval, object detection, and
  segmentation. Moreover, SPARC improves model faithfulness and captioning in
  foundational vision-language models.
paper_ai_summary: SPARC pretrains fine-grained multimodal representations using sparse contrastive alignment, improving both global and local information in vision-language models and enhancing performance on classification and region-level tasks.
paper_ai_keywords: SPARC fine-grained multimodal representations image-text pairs image patches language tokens sparse similarity metric vision embedding fine-grained sequence-wise loss contrastive loss global image embeddings text embeddings faithfulness captioning