🚀 First Benchmark of 
text
🚀 First Benchmark of 
@OpenAI
mention
OpenAI
's 4o Image Generation Model!
text
's 4o Image Generation Model!
new_line
new_line
We've just completed the first-ever (to our knowledge) benchmarking of the new OpenAI 4o image generation model, and the results are impressive!
text
We've just completed the first-ever (to our knowledge) benchmarking of the new OpenAI 4o image generation model, and the results are impressive!
new_line
new_line
In our tests, OpenAI 4o image generation absolutely crushed  leading competitors, including 
text
In our tests, OpenAI 4o image generation absolutely crushed  leading competitors, including 
@black-forest-labs
mention
black-forest-labs
, 
text
, 
@google
mention
google
, 
text
, 
@xai-org
mention
xai-org
, Ideogram, Recraft, and 
text
, Ideogram, Recraft, and 
@deepseek-ai
mention
deepseek-ai
, in prompt alignment and coherence! They hold a gap of more than 20% to the nearest competitor in terms of Bradley-Terry score, the biggest we have seen since the beginning of the benchmark!
text
, in prompt alignment and coherence! They hold a gap of more than 20% to the nearest competitor in terms of Bradley-Terry score, the biggest we have seen since the beginning of the benchmark!
new_line
new_line
The benchmarks are based on 200k human responses collected through our API. However, the most challenging part wasn't the benchmarking itself, but generating and downloading the images:
text
The benchmarks are based on 200k human responses collected through our API. However, the most challenging part wasn't the benchmarking itself, but generating and downloading the images:
new_line
new_line
- 5 hours to generate 1000 images (no API available yet)
text
- 5 hours to generate 1000 images (no API available yet)
new_line
- Just 10 minutes to set up and launch the benchmark
text
- Just 10 minutes to set up and launch the benchmark
new_line
- Over 200,000 responses rapidly collected
text
- Over 200,000 responses rapidly collected
new_line
new_line
While generating the images, we faced some hurdles that meant that we had to leave out certain parts of our prompt set. Particularly we observed that the  OpenAI 4o model proactively refused to generate certain images:
text
While generating the images, we faced some hurdles that meant that we had to leave out certain parts of our prompt set. Particularly we observed that the  OpenAI 4o model proactively refused to generate certain images:
new_line
new_line
🚫 Styles of living artists: completely blocked
text
🚫 Styles of living artists: completely blocked
new_line
🚫 Copyrighted characters (e.g., Darth Vader, Pokémon): initially generated but subsequently blocked
text
🚫 Copyrighted characters (e.g., Darth Vader, Pokémon): initially generated but subsequently blocked
new_line
new_line
Overall, OpenAI 4o stands out significantly in alignment and coherence, especially excelling in certain unusual prompts that have historically caused issues such as: 'A chair on a cat.' See the images for more examples!
text
Overall, OpenAI 4o stands out significantly in alignment and coherence, especially excelling in certain unusual prompts that have historically caused issues such as: 'A chair on a cat.' See the images for more examples!
new_line
🚀 First Benchmark of @OpenAI's 4o Image Generation Model!

We've just completed the first-ever (to our knowledge) benchmarking of the new OpenAI 4o image generation model, and the results are impressive!

In our tests, OpenAI 4o image generation absolutely crushed  leading competitors, including @black-forest-labs, @google, @xai-org, Ideogram, Recraft, and @deepseek-ai, in prompt alignment and coherence! They hold a gap of more than 20% to the nearest competitor in terms of Bradley-Terry score, the biggest we have seen since the beginning of the benchmark!

The benchmarks are based on 200k human responses collected through our API. However, the most challenging part wasn't the benchmarking itself, but generating and downloading the images:

- 5 hours to generate 1000 images (no API available yet)
- Just 10 minutes to set up and launch the benchmark
- Over 200,000 responses rapidly collected

While generating the images, we faced some hurdles that meant that we had to leave out certain parts of our prompt set. Particularly we observed that the  OpenAI 4o model proactively refused to generate certain images:

🚫 Styles of living artists: completely blocked
🚫 Copyrighted characters (e.g., Darth Vader, Pokémon): initially generated but subsequently blocked

Overall, OpenAI 4o stands out significantly in alignment and coherence, especially excelling in certain unusual prompts that have historically caused issues such as: 'A chair on a cat.' See the images for more examples!

🔥
tmanuel maalber Kchanger jasoncorkill LinoGiger yohoji LucStr John6666 windmgc daniilpyatko risxyie Arcc99 xanzzzz spacegrey23 Takugen murphz dantezxcd
👍
tmanuel Kchanger jasoncorkill framefactory daniilpyatko taylores murphz dantezxcd on3dj
🚀
maalber Kchanger jasoncorkill LinoGiger daniilpyatko neurlang dantezxcd
👀
maalber murphz dantezxcd
66f5624c42b853e73e0738eb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/qW3-oKDLFJpue2iS5VjT2.jpeg
Jason Corkill
jasoncorkill
user