title: An Early Evaluation of GPT-4V(ision)
summary: In this paper, we evaluate different abilities of GPT-4V including visual
  understanding, language understanding, visual puzzle solving, and understanding
  of other modalities such as depth, thermal, video, and audio. To estimate
  GPT-4V's performance, we manually construct 656 test instances and carefully
  evaluate the results of GPT-4V. The highlights of our findings are as follows:
  (1) GPT-4V exhibits impressive performance on English visual-centric benchmarks
  but fails to recognize simple Chinese texts in the images; (2) GPT-4V shows
  inconsistent refusal behavior when answering questions related to sensitive
  traits such as gender, race, and age; (3) GPT-4V obtains worse results than
  GPT-4 (API) on language understanding tasks including general language
  understanding benchmarks and visual commonsense knowledge evaluation
  benchmarks; (4) Few-shot prompting can improve GPT-4V's performance on both
  visual understanding and language understanding; (5) GPT-4V struggles to find
  the nuances between two similar images and solve the easy math picture puzzles;
  (6) GPT-4V shows non-trivial performance on the tasks of similar modalities to
  image, such as video and thermal. Our experimental results reveal the ability
  and limitations of GPT-4V and we hope our paper can provide some insights into
  the application and research of GPT-4V.
paper_authors._id: 6539d0bf2a101a151db0bb20
paper_authors.name: Yang Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-26T08:05:51.960Z
paper_authors._id: 64f19725a5ca432503cae323
paper_authors.avatarUrl: /avatars/39c82fd3f831633a46e7fceb7580f362.svg
paper_authors.fullname: Albert
paper_authors.type: user
paper_authors.user: AlbertWu
paper_authors._id: 6539d0bf2a101a151db0bb21
paper_authors.name: Shilong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T07:55:43.914Z
paper_authors._id: 626686254e996ebbb6d783b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650886169921-noauth.jpeg
paper_authors.fullname: wangshilong
paper_authors.type: user
paper_authors.user: shilong
paper_authors._id: 6539d0bf2a101a151db0bb22
paper_authors.name: Hao Yang
paper_authors._id: 6539d0bf2a101a151db0bb23
paper_authors.name: Tian Zheng
paper_authors._id: 6539d0bf2a101a151db0bb24
paper_authors.name: Hongbo Zhang
paper_authors._id: 6539d0bf2a101a151db0bb25
paper_authors.name: Yanyan Zhao
paper_authors._id: 6539d0bf2a101a151db0bb26
paper_authors.name: Bing Qin
paper_title: An Early Evaluation of GPT-4V(ision)
paper_summary: In this paper, we evaluate different abilities of GPT-4V including visual
  understanding, language understanding, visual puzzle solving, and understanding
  of other modalities such as depth, thermal, video, and audio. To estimate
  GPT-4V's performance, we manually construct 656 test instances and carefully
  evaluate the results of GPT-4V. The highlights of our findings are as follows:
  (1) GPT-4V exhibits impressive performance on English visual-centric benchmarks
  but fails to recognize simple Chinese texts in the images; (2) GPT-4V shows
  inconsistent refusal behavior when answering questions related to sensitive
  traits such as gender, race, and age; (3) GPT-4V obtains worse results than
  GPT-4 (API) on language understanding tasks including general language
  understanding benchmarks and visual commonsense knowledge evaluation
  benchmarks; (4) Few-shot prompting can improve GPT-4V's performance on both
  visual understanding and language understanding; (5) GPT-4V struggles to find
  the nuances between two similar images and solve the easy math picture puzzles;
  (6) GPT-4V shows non-trivial performance on the tasks of similar modalities to
  image, such as video and thermal. Our experimental results reveal the ability
  and limitations of GPT-4V and we hope our paper can provide some insights into
  the application and research of GPT-4V.
paper_ai_keywords: GPT-4V visual understanding language understanding visual puzzle solving processing depth thermal imaging video analysis audio processing manual construction test instances benchmark evaluation English visual-centric benchmarks Chinese text recognition refusal behavior sensitive traits general language understanding benchmarks visual commonsense knowledge few-shot prompting image similarity analysis easy math picture puzzles thermal imaging tasks video task performance