SITTA: A Semantic Image-Text Alignment for Image Captioning
Textual and semantic comprehension of images is essential for generating
proper captions. The comprehension requires detection of objects, modeling of
relations between them, an assessment of the semantics of the scene and,
finally, representing the extracted knowledge in a language space. To achieve
rich language capabilities while ensuring good image-language mappings,
pretrained language models (LMs) were conditioned on pretrained multi-modal
(image-text) models that allow for image inputs. This requires an alignment of
the image representation of the multi-modal model with the language
representations of a generative LM. However, it is not clear how to best
transfer semantics detected by the vision encoder of the multi-modal model to
the LM. We introduce two novel ways of constructing a linear mapping that
successfully transfers semantics between the embedding spaces of the two
pretrained models. The first aligns the embedding space of the multi-modal
language encoder with the embedding space of the pretrained LM via token
correspondences. The latter leverages additional data that consists of
image-text pairs to construct the mapping directly from vision to language
space. Using our semantic mappings, we unlock image captioning for LMs without
access to gradient information. By using different sources of data we achieve
strong captioning performance on MS-COCO and Flickr30k datasets. Even in the
face of limited data, our method partly exceeds the performance of other
zero-shot and even finetuned competitors. Our ablation studies show that even
LMs at a scale of merely 250M parameters can generate decent captions employing
our semantic mappings. Our approach makes image captioning more accessible for
institutions with restricted computational resources.
64af59b9a609b29cc7b240bd
Fabian Paischer
admin_assigned
2023-07-13T09:28:18.071Z
648826f845a9218318e0272c
https://cdn-avatars.huggingface.co/v1/production/uploads/648826f845a9218318e0272c/1zUQFA7TdQ8WC9Chskx6b.jpeg
Fabian Paischer
user
paischer101
64af59b9a609b29cc7b240be
Thomas Adler
64af59b9a609b29cc7b240bf
Markus Hofmarcher
64af59b9a609b29cc7b240c0
Sepp Hochreiter
SITTA: A Semantic Image-Text Alignment for Image Captioning
Textual and semantic comprehension of images is essential for generating
proper captions. The comprehension requires detection of objects, modeling of
relations between them, an assessment of the semantics of the scene and,
finally, representing the extracted knowledge in a language space. To achieve
rich language capabilities while ensuring good image-language mappings,
pretrained language models (LMs) were conditioned on pretrained multi-modal
(image-text) models that allow for image inputs. This requires an alignment of
the image representation of the multi-modal model with the language
representations of a generative LM. However, it is not clear how to best
transfer semantics detected by the vision encoder of the multi-modal model to
the LM. We introduce two novel ways of constructing a linear mapping that
successfully transfers semantics between the embedding spaces of the two
pretrained models. The first aligns the embedding space of the multi-modal
language encoder with the embedding space of the pretrained LM via token
correspondences. The latter leverages additional data that consists of
image-text pairs to construct the mapping directly from vision to language
space. Using our semantic mappings, we unlock image captioning for LMs without
access to gradient information. By using different sources of data we achieve
strong captioning performance on MS-COCO and Flickr30k datasets. Even in the
face of limited data, our method partly exceeds the performance of other
zero-shot and even finetuned competitors. Our ablation studies show that even
LMs at a scale of merely 250M parameters can generate decent captions employing
our semantic mappings. Our approach makes image captioning more accessible for
institutions with restricted computational resources.
The paper presents novel methods for constructing semantic mappings between image and language embedding spaces, enabling effective image captioning without extensive gradient information.
pretrained language models multi-modal models token correspondences image captioning MS-COCO Flickr30k zero-shot parameter-efficient fine-tuning semantic mappings