title: SITTA: A Semantic Image-Text Alignment for Image Captioning
summary: Textual and semantic comprehension of images is essential for generating
  proper captions. The comprehension requires detection of objects, modeling of
  relations between them, an assessment of the semantics of the scene and,
  finally, representing the extracted knowledge in a language space. To achieve
  rich language capabilities while ensuring good image-language mappings,
  pretrained language models (LMs) were conditioned on pretrained multi-modal
  (image-text) models that allow for image inputs. This requires an alignment of
  the image representation of the multi-modal model with the language
  representations of a generative LM. However, it is not clear how to best
  transfer semantics detected by the vision encoder of the multi-modal model to
  the LM. We introduce two novel ways of constructing a linear mapping that
  successfully transfers semantics between the embedding spaces of the two
  pretrained models. The first aligns the embedding space of the multi-modal
  language encoder with the embedding space of the pretrained LM via token
  correspondences. The latter leverages additional data that consists of
  image-text pairs to construct the mapping directly from vision to language
  space. Using our semantic mappings, we unlock image captioning for LMs without
  access to gradient information. By using different sources of data we achieve
  strong captioning performance on MS-COCO and Flickr30k datasets. Even in the
  face of limited data, our method partly exceeds the performance of other
  zero-shot and even finetuned competitors. Our ablation studies show that even
  LMs at a scale of merely 250M parameters can generate decent captions employing
  our semantic mappings. Our approach makes image captioning more accessible for
  institutions with restricted computational resources.
paper_authors._id: 64af59b9a609b29cc7b240bd
paper_authors.name: Fabian Paischer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:28:18.071Z
paper_authors._id: 648826f845a9218318e0272c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648826f845a9218318e0272c/1zUQFA7TdQ8WC9Chskx6b.jpeg
paper_authors.fullname: Fabian Paischer
paper_authors.type: user
paper_authors.user: paischer101
paper_authors._id: 64af59b9a609b29cc7b240be
paper_authors.name: Thomas Adler
paper_authors._id: 64af59b9a609b29cc7b240bf
paper_authors.name: Markus Hofmarcher
paper_authors._id: 64af59b9a609b29cc7b240c0
paper_authors.name: Sepp Hochreiter
paper_title: SITTA: A Semantic Image-Text Alignment for Image Captioning
paper_summary: Textual and semantic comprehension of images is essential for generating
  proper captions. The comprehension requires detection of objects, modeling of
  relations between them, an assessment of the semantics of the scene and,
  finally, representing the extracted knowledge in a language space. To achieve
  rich language capabilities while ensuring good image-language mappings,
  pretrained language models (LMs) were conditioned on pretrained multi-modal
  (image-text) models that allow for image inputs. This requires an alignment of
  the image representation of the multi-modal model with the language
  representations of a generative LM. However, it is not clear how to best
  transfer semantics detected by the vision encoder of the multi-modal model to
  the LM. We introduce two novel ways of constructing a linear mapping that
  successfully transfers semantics between the embedding spaces of the two
  pretrained models. The first aligns the embedding space of the multi-modal
  language encoder with the embedding space of the pretrained LM via token
  correspondences. The latter leverages additional data that consists of
  image-text pairs to construct the mapping directly from vision to language
  space. Using our semantic mappings, we unlock image captioning for LMs without
  access to gradient information. By using different sources of data we achieve
  strong captioning performance on MS-COCO and Flickr30k datasets. Even in the
  face of limited data, our method partly exceeds the performance of other
  zero-shot and even finetuned competitors. Our ablation studies show that even
  LMs at a scale of merely 250M parameters can generate decent captions employing
  our semantic mappings. Our approach makes image captioning more accessible for
  institutions with restricted computational resources.
paper_ai_keywords: pretrained language models (LMs) multi-modal (image-text) models vision encoders embedding spaces token correspondences semantic mappings zero-shot ablation studies