On the Limitations of Vision-Language Models in Understanding Image
  Transforms
Vision Language Models (VLMs) have demonstrated significant potential in
various downstream tasks, including Image/Video Generation, Visual Question
Answering, Multimodal Chatbots, and Video Understanding. However, these models
often struggle with basic image transformations. This paper investigates the
image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by
Google. Our findings reveal that these models lack comprehension of multiple
image-level augmentations. To facilitate this study, we created an augmented
version of the Flickr8k dataset, pairing each image with a detailed description
of the applied transformation. We further explore how this deficiency impacts
downstream tasks, particularly in image editing, and evaluate the performance
of state-of-the-art Image2Image models on simple transformations.
67d4194fc9f149faff46913c
Ahmad Mustafa Anis
claimed_verified
2025-03-16T21:14:01.381Z
6246908d8031dcfa9ef6d80b
https://cdn-avatars.huggingface.co/v1/production/uploads/6246908d8031dcfa9ef6d80b/hVdURjUl1RS2MZf4qOhvI.jpeg
Ahmad Mustafa Anis
user
AhmadMustafa
67d4194fc9f149faff46913d
Hasnain Ali
67d4194fc9f149faff46913e
Saquib Sarfraz
On the Limitations of Vision-Language Models in Understanding Image
  Transforms
Vision Language Models (VLMs) have demonstrated significant potential in
various downstream tasks, including Image/Video Generation, Visual Question
Answering, Multimodal Chatbots, and Video Understanding. However, these models
often struggle with basic image transformations. This paper investigates the
image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by
Google. Our findings reveal that these models lack comprehension of multiple
image-level augmentations. To facilitate this study, we created an augmented
version of the Flickr8k dataset, pairing each image with a detailed description
of the applied transformation. We further explore how this deficiency impacts
downstream tasks, particularly in image editing, and evaluate the performance
of state-of-the-art Image2Image models on simple transformations.
Studies on Vision Language Models, such as CLIP and SigLIP, show poor understanding of image augmentations, impacting tasks like image editing, as evidenced by evaluations on modified Image2Image models.
Vision Language Models CLIP SigLIP image-level augmentations downstream tasks image editing Image2Image models