title: GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image
    Generation
summary: The recent breakthroughs in OpenAI's GPT4o model have demonstrated
  surprisingly good capabilities in image generation and editing, resulting in
  significant excitement in the community. This technical report presents the
  first-look evaluation benchmark (named GPT-ImgEval), quantitatively and
  qualitatively diagnosing GPT-4o's performance across three critical dimensions:
  (1) generation quality, (2) editing proficiency, and (3) world
  knowledge-informed semantic synthesis. Across all three tasks, GPT-4o
  demonstrates strong performance, significantly surpassing existing methods in
  both image generation control and output quality, while also showcasing
  exceptional knowledge reasoning capabilities. Furthermore, based on the
  GPT-4o's generated data, we propose a classification-model-based approach to
  investigate the underlying architecture of GPT-4o, where our empirical results
  suggest the model consists of an auto-regressive (AR) combined with a
  diffusion-based head for image decoding, rather than the VAR-like
  architectures. We also provide a complete speculation on GPT-4o's overall
  architecture. In addition, we conduct a series of analyses to identify and
  visualize GPT-4o's specific limitations and the synthetic artifacts commonly
  observed in its image generation. We also present a comparative study of
  multi-round image editing between GPT-4o and Gemini 2.0 Flash, and discuss the
  safety implications of GPT-4o's outputs, particularly their detectability by
  existing image forensic models. We hope that our work can offer valuable
  insight and provide a reliable benchmark to guide future research, foster
  reproducibility, and accelerate innovation in the field of image generation and
  beyond. The codes and datasets used for evaluating GPT-4o can be found at
  https://github.com/PicoTrex/GPT-ImgEval.
paper_authors._id: 67ef502ce803d818f00e1b94
paper_authors.name: Zhiyuan Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:56:10.006Z
paper_authors._id: 67dd44d52599dbcecfb4cb9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/9yZaPuSMY-evu25DPT0o5.png
paper_authors.fullname: Zhiyuan Yan
paper_authors.type: user
paper_authors.user: zhiyuanyan1
paper_authors._id: 67ef502ce803d818f00e1b95
paper_authors.name: Junyan Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:17:10.032Z
paper_authors._id: 66978ee0b8656f6506b4acb2
paper_authors.avatarUrl: /avatars/298acb8222e189fce4368985ee5374a1.svg
paper_authors.fullname: Junyan Ye
paper_authors.type: user
paper_authors.user: Yejy53
paper_authors._id: 67ef502ce803d818f00e1b96
paper_authors.name: Weijia Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:16:44.819Z
paper_authors._id: 66d5b56c77a026c3d2086a79
paper_authors.avatarUrl: /avatars/45da07fd82fd455955faa05b27a6393f.svg
paper_authors.fullname: Weijia Li
paper_authors.type: user
paper_authors.user: liweijia
paper_authors._id: 67ef502ce803d818f00e1b97
paper_authors.name: Zilong Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:56.501Z
paper_authors._id: 6487e158f675b4a7867f45fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6487e158f675b4a7867f45fa/J0sls6zZ682o-SH7iQs7B.jpeg
paper_authors.fullname: Zilong Huang
paper_authors.type: user
paper_authors.user: SereinH
paper_authors._id: 67ef502ce803d818f00e1b98
paper_authors.name: Shenghai Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:09:05.246Z
paper_authors._id: 63468720dd6d90d82ccf3450
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63468720dd6d90d82ccf3450/tVBFlmZNz8FRMkOrDaDID.jpeg
paper_authors.fullname: YSH
paper_authors.type: user
paper_authors.user: BestWishYsh
paper_authors._id: 67ef502ce803d818f00e1b99
paper_authors.name: Xiangyang He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:09:02.573Z
paper_authors._id: 67ef53656c7ba428e7c2e605
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/fFVJTpMRKF15Bf63yZEG_.png
paper_authors.fullname: He
paper_authors.type: user
paper_authors.user: shawnxyh
paper_authors._id: 67ef502ce803d818f00e1b9a
paper_authors.name: Kaiqing Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:16:28.452Z
paper_authors._id: 6459a47e4fe72fae522b4fc9
paper_authors.avatarUrl: /avatars/a4139f8e348081e45b28dd99d96908d3.svg
paper_authors.fullname: Kaiqing.Lin
paper_authors.type: user
paper_authors.user: lin6123
paper_authors._id: 67ef502ce803d818f00e1b9b
paper_authors.name: Jun He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:59.877Z
paper_authors._id: 670ddb69d6ac6394419d88c5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/XxnGNaX3FWug4aiZVjg93.png
paper_authors.fullname: Jun He
paper_authors.type: user
paper_authors.user: JunHe0915
paper_authors._id: 67ef502ce803d818f00e1b9c
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:17:18.123Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 67ef502ce803d818f00e1b9d
paper_authors.name: Li Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:40.281Z
paper_authors._id: 66135a5e50350afe76beebce
paper_authors.avatarUrl: /avatars/370a4b83949355feb050c2cb0425c264.svg
paper_authors.fullname: yl2488
paper_authors.type: user
paper_authors.user: yl2488
paper_title: GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image
    Generation
paper_summary: The recent breakthroughs in OpenAI's GPT4o model have demonstrated
  surprisingly good capabilities in image generation and editing, resulting in
  significant excitement in the community. This technical report presents the
  first-look evaluation benchmark (named GPT-ImgEval), quantitatively and
  qualitatively diagnosing GPT-4o's performance across three critical dimensions:
  (1) generation quality, (2) editing proficiency, and (3) world
  knowledge-informed semantic synthesis. Across all three tasks, GPT-4o
  demonstrates strong performance, significantly surpassing existing methods in
  both image generation control and output quality, while also showcasing
  exceptional knowledge reasoning capabilities. Furthermore, based on the
  GPT-4o's generated data, we propose a classification-model-based approach to
  investigate the underlying architecture of GPT-4o, where our empirical results
  suggest the model consists of an auto-regressive (AR) combined with a
  diffusion-based head for image decoding, rather than the VAR-like
  architectures. We also provide a complete speculation on GPT-4o's overall
  architecture. In addition, we conduct a series of analyses to identify and
  visualize GPT-4o's specific limitations and the synthetic artifacts commonly
  observed in its image generation. We also present a comparative study of
  multi-round image editing between GPT-4o and Gemini 2.0 Flash, and discuss the
  safety implications of GPT-4o's outputs, particularly their detectability by
  existing image forensic models. We hope that our work can offer valuable
  insight and provide a reliable benchmark to guide future research, foster
  reproducibility, and accelerate innovation in the field of image generation and
  beyond. The codes and datasets used for evaluating GPT-4o can be found at
  https://github.com/PicoTrex/GPT-ImgEval.
paper_ai_summary: GPT-4o demonstrates strong image generation and editing capabilities with a proposed auto-regressive and diffusion-based architecture, surpassing existing methods and highlighting specific limitations and safety concerns.
paper_ai_keywords: auto-regressive diffusion-based multi-round image editing Gemiini 2.0 Flash image forensic models