An Early Evaluation of GPT-4V(ision)
In this paper, we evaluate different abilities of GPT-4V including visual
understanding, language understanding, visual puzzle solving, and understanding
of other modalities such as depth, thermal, video, and audio. To estimate
GPT-4V's performance, we manually construct 656 test instances and carefully
evaluate the results of GPT-4V. The highlights of our findings are as follows:
(1) GPT-4V exhibits impressive performance on English visual-centric benchmarks
but fails to recognize simple Chinese texts in the images; (2) GPT-4V shows
inconsistent refusal behavior when answering questions related to sensitive
traits such as gender, race, and age; (3) GPT-4V obtains worse results than
GPT-4 (API) on language understanding tasks including general language
understanding benchmarks and visual commonsense knowledge evaluation
benchmarks; (4) Few-shot prompting can improve GPT-4V's performance on both
visual understanding and language understanding; (5) GPT-4V struggles to find
the nuances between two similar images and solve the easy math picture puzzles;
(6) GPT-4V shows non-trivial performance on the tasks of similar modalities to
image, such as video and thermal. Our experimental results reveal the ability
and limitations of GPT-4V and we hope our paper can provide some insights into
the application and research of GPT-4V.
6539d0bf2a101a151db0bb20
Yang Wu
claimed_verified
2023-10-26T08:05:51.960Z
64f19725a5ca432503cae323
/avatars/39c82fd3f831633a46e7fceb7580f362.svg
Albert
user
AlbertWu
6539d0bf2a101a151db0bb21
Shilong Wang
admin_assigned
2023-10-26T07:55:43.914Z
626686254e996ebbb6d783b2
https://cdn-avatars.huggingface.co/v1/production/uploads/1650886169921-noauth.jpeg
wangshilong
user
shilong
6539d0bf2a101a151db0bb22
Hao Yang
6539d0bf2a101a151db0bb23
Tian Zheng
6539d0bf2a101a151db0bb24
Hongbo Zhang
6539d0bf2a101a151db0bb25
Yanyan Zhao
6539d0bf2a101a151db0bb26
Bing Qin
An Early Evaluation of GPT-4V(ision)
In this paper, we evaluate different abilities of GPT-4V including visual
understanding, language understanding, visual puzzle solving, and understanding
of other modalities such as depth, thermal, video, and audio. To estimate
GPT-4V's performance, we manually construct 656 test instances and carefully
evaluate the results of GPT-4V. The highlights of our findings are as follows:
(1) GPT-4V exhibits impressive performance on English visual-centric benchmarks
but fails to recognize simple Chinese texts in the images; (2) GPT-4V shows
inconsistent refusal behavior when answering questions related to sensitive
traits such as gender, race, and age; (3) GPT-4V obtains worse results than
GPT-4 (API) on language understanding tasks including general language
understanding benchmarks and visual commonsense knowledge evaluation
benchmarks; (4) Few-shot prompting can improve GPT-4V's performance on both
visual understanding and language understanding; (5) GPT-4V struggles to find
the nuances between two similar images and solve the easy math picture puzzles;
(6) GPT-4V shows non-trivial performance on the tasks of similar modalities to
image, such as video and thermal. Our experimental results reveal the ability
and limitations of GPT-4V and we hope our paper can provide some insights into
the application and research of GPT-4V.
GPT-4V demonstrates strong visual understanding but has limitations in language comprehension, handling sensitive data, modalities like depth and audio, and fine visual nuances.
GPT-4V visual understanding language understanding visual puzzle solving depth thermal video audio few-shot prompting visual commonsense knowledge