GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image
  Generation
The recent breakthroughs in OpenAI's GPT4o model have demonstrated
surprisingly good capabilities in image generation and editing, resulting in
significant excitement in the community. This technical report presents the
first-look evaluation benchmark (named GPT-ImgEval), quantitatively and
qualitatively diagnosing GPT-4o's performance across three critical dimensions:
(1) generation quality, (2) editing proficiency, and (3) world
knowledge-informed semantic synthesis. Across all three tasks, GPT-4o
demonstrates strong performance, significantly surpassing existing methods in
both image generation control and output quality, while also showcasing
exceptional knowledge reasoning capabilities. Furthermore, based on the
GPT-4o's generated data, we propose a classification-model-based approach to
investigate the underlying architecture of GPT-4o, where our empirical results
suggest the model consists of an auto-regressive (AR) combined with a
diffusion-based head for image decoding, rather than the VAR-like
architectures. We also provide a complete speculation on GPT-4o's overall
architecture. In addition, we conduct a series of analyses to identify and
visualize GPT-4o's specific limitations and the synthetic artifacts commonly
observed in its image generation. We also present a comparative study of
multi-round image editing between GPT-4o and Gemini 2.0 Flash, and discuss the
safety implications of GPT-4o's outputs, particularly their detectability by
existing image forensic models. We hope that our work can offer valuable
insight and provide a reliable benchmark to guide future research, foster
reproducibility, and accelerate innovation in the field of image generation and
beyond. The codes and datasets used for evaluating GPT-4o can be found at
https://github.com/PicoTrex/GPT-ImgEval.
67ef502ce803d818f00e1b94
Zhiyuan Yan
claimed_verified
2025-04-08T06:56:10.006Z
67dd44d52599dbcecfb4cb9c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/9yZaPuSMY-evu25DPT0o5.png
Zhiyuan Yan
user
zhiyuanyan1
67ef502ce803d818f00e1b95
Junyan Ye
admin_assigned
2025-04-04T07:17:10.032Z
66978ee0b8656f6506b4acb2
/avatars/298acb8222e189fce4368985ee5374a1.svg
Junyan Ye
user
Yejy53
67ef502ce803d818f00e1b96
Weijia Li
admin_assigned
2025-04-04T07:16:44.819Z
66d5b56c77a026c3d2086a79
/avatars/45da07fd82fd455955faa05b27a6393f.svg
Weijia Li
user
liweijia
67ef502ce803d818f00e1b97
Zilong Huang
claimed_verified
2025-04-04T07:08:56.501Z
6487e158f675b4a7867f45fa
https://cdn-avatars.huggingface.co/v1/production/uploads/6487e158f675b4a7867f45fa/J0sls6zZ682o-SH7iQs7B.jpeg
Zilong Huang
user
SereinH
67ef502ce803d818f00e1b98
Shenghai Yuan
claimed_verified
2025-04-04T07:09:05.246Z
63468720dd6d90d82ccf3450
https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
YSH
user
BestWishYsh
67ef502ce803d818f00e1b99
Xiangyang He
claimed_verified
2025-04-04T07:09:02.573Z
67ef53656c7ba428e7c2e605
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/fFVJTpMRKF15Bf63yZEG_.png
He
user
shawnxyh
67ef502ce803d818f00e1b9a
Kaiqing Lin
admin_assigned
2025-04-04T07:16:28.452Z
6459a47e4fe72fae522b4fc9
/avatars/a4139f8e348081e45b28dd99d96908d3.svg
Kaiqing.Lin
user
lin6123
67ef502ce803d818f00e1b9b
Jun He
claimed_verified
2025-04-04T07:08:59.877Z
670ddb69d6ac6394419d88c5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/XxnGNaX3FWug4aiZVjg93.png
Jun He
user
JunHe0915
67ef502ce803d818f00e1b9c
Conghui He
admin_assigned
2025-04-04T07:17:18.123Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
67ef502ce803d818f00e1b9d
Li Yuan
claimed_verified
2025-04-04T07:08:40.281Z
66135a5e50350afe76beebce
/avatars/370a4b83949355feb050c2cb0425c264.svg
yl2488
user
yl2488
GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image
  Generation
The recent breakthroughs in OpenAI's GPT4o model have demonstrated
surprisingly good capabilities in image generation and editing, resulting in
significant excitement in the community. This technical report presents the
first-look evaluation benchmark (named GPT-ImgEval), quantitatively and
qualitatively diagnosing GPT-4o's performance across three critical dimensions:
(1) generation quality, (2) editing proficiency, and (3) world
knowledge-informed semantic synthesis. Across all three tasks, GPT-4o
demonstrates strong performance, significantly surpassing existing methods in
both image generation control and output quality, while also showcasing
exceptional knowledge reasoning capabilities. Furthermore, based on the
GPT-4o's generated data, we propose a classification-model-based approach to
investigate the underlying architecture of GPT-4o, where our empirical results
suggest the model consists of an auto-regressive (AR) combined with a
diffusion-based head for image decoding, rather than the VAR-like
architectures. We also provide a complete speculation on GPT-4o's overall
architecture. In addition, we conduct a series of analyses to identify and
visualize GPT-4o's specific limitations and the synthetic artifacts commonly
observed in its image generation. We also present a comparative study of
multi-round image editing between GPT-4o and Gemini 2.0 Flash, and discuss the
safety implications of GPT-4o's outputs, particularly their detectability by
existing image forensic models. We hope that our work can offer valuable
insight and provide a reliable benchmark to guide future research, foster
reproducibility, and accelerate innovation in the field of image generation and
beyond. The codes and datasets used for evaluating GPT-4o can be found at
https://github.com/PicoTrex/GPT-ImgEval.
GPT-4o demonstrates strong image generation and editing capabilities with a proposed auto-regressive and diffusion-based architecture, surpassing existing methods and highlighting specific limitations and safety concerns.
auto-regressive diffusion-based multi-round image editing Gemiini 2.0 Flash image forensic models