Rapidata/text-2-video-human-preferences-pika2.2
Viewer
•
Updated
•
1.68k
•
12
•
4
Manuel
tmanuel
AI & ML interests
None yet
Recent Activity
liked
a dataset
about 2 hours ago
Rapidata/text-2-video-human-preferences-pika2.2
liked
a dataset
about 2 hours ago
Rapidata/Reve-AI-Halfmoon_t2i_human_preference
reacted
to
jasoncorkill's
post
with 🔥
5 days ago
🚀 Rapidata: Setting the Standard for Model Evaluation
Rapidata is proud to announce our first independent appearance in academic research, featured in the Lumina-Image 2.0 paper. This marks the beginning of our journey to become the standard for testing text-to-image and generative models. Our expertise in large-scale human annotations allows researchers to refine their models with accurate, real-world feedback.
As we continue to establish ourselves as a key player in model evaluation, we’re here to support researchers with high-quality annotations at scale. Reach out to [email protected] to see how we can help.
https://huggingface.co/papers/2503.21758
Organizations
tmanuel's activity
liked
2
datasets
about 2 hours ago
reacted to
jasoncorkill's
post with 🔥
5 days ago
Post
2332
🚀 Rapidata: Setting the Standard for Model Evaluation
Rapidata is proud to announce our first independent appearance in academic research, featured in the Lumina-Image 2.0 paper. This marks the beginning of our journey to become the standard for testing text-to-image and generative models. Our expertise in large-scale human annotations allows researchers to refine their models with accurate, real-world feedback.
As we continue to establish ourselves as a key player in model evaluation, we’re here to support researchers with high-quality annotations at scale. Reach out to [email protected] to see how we can help.
Lumina-Image 2.0: A Unified and Efficient Image Generative Framework (2503.21758)
Rapidata is proud to announce our first independent appearance in academic research, featured in the Lumina-Image 2.0 paper. This marks the beginning of our journey to become the standard for testing text-to-image and generative models. Our expertise in large-scale human annotations allows researchers to refine their models with accurate, real-world feedback.
As we continue to establish ourselves as a key player in model evaluation, we’re here to support researchers with high-quality annotations at scale. Reach out to [email protected] to see how we can help.
Lumina-Image 2.0: A Unified and Efficient Image Generative Framework (2503.21758)
reacted to
jasoncorkill's
post with 🔥
11 days ago
Post
2235
🔥 It's out! We published the dataset for our evaluation of
@OpenAI
's new 4o image generation model.
Rapidata/OpenAI-4o_t2i_human_preference
Yesterday we published the first large evaluation of the new model, showing that it absolutely leaves the competition in the dust. We have now made the results and data available here! Please check it out and ❤️ !
Rapidata/OpenAI-4o_t2i_human_preference
Yesterday we published the first large evaluation of the new model, showing that it absolutely leaves the competition in the dust. We have now made the results and data available here! Please check it out and ❤️ !
liked
3
datasets
12 days ago
reacted to
jasoncorkill's
post with 🔥👍
12 days ago
Post
2025
🚀 First Benchmark of
@OpenAI
's 4o Image Generation Model!
We've just completed the first-ever (to our knowledge) benchmarking of the new OpenAI 4o image generation model, and the results are impressive!
In our tests, OpenAI 4o image generation absolutely crushed leading competitors, including @black-forest-labs , @google , @xai-org , Ideogram, Recraft, and @deepseek-ai , in prompt alignment and coherence! They hold a gap of more than 20% to the nearest competitor in terms of Bradley-Terry score, the biggest we have seen since the beginning of the benchmark!
The benchmarks are based on 200k human responses collected through our API. However, the most challenging part wasn't the benchmarking itself, but generating and downloading the images:
- 5 hours to generate 1000 images (no API available yet)
- Just 10 minutes to set up and launch the benchmark
- Over 200,000 responses rapidly collected
While generating the images, we faced some hurdles that meant that we had to leave out certain parts of our prompt set. Particularly we observed that the OpenAI 4o model proactively refused to generate certain images:
🚫 Styles of living artists: completely blocked
🚫 Copyrighted characters (e.g., Darth Vader, Pokémon): initially generated but subsequently blocked
Overall, OpenAI 4o stands out significantly in alignment and coherence, especially excelling in certain unusual prompts that have historically caused issues such as: 'A chair on a cat.' See the images for more examples!
We've just completed the first-ever (to our knowledge) benchmarking of the new OpenAI 4o image generation model, and the results are impressive!
In our tests, OpenAI 4o image generation absolutely crushed leading competitors, including @black-forest-labs , @google , @xai-org , Ideogram, Recraft, and @deepseek-ai , in prompt alignment and coherence! They hold a gap of more than 20% to the nearest competitor in terms of Bradley-Terry score, the biggest we have seen since the beginning of the benchmark!
The benchmarks are based on 200k human responses collected through our API. However, the most challenging part wasn't the benchmarking itself, but generating and downloading the images:
- 5 hours to generate 1000 images (no API available yet)
- Just 10 minutes to set up and launch the benchmark
- Over 200,000 responses rapidly collected
While generating the images, we faced some hurdles that meant that we had to leave out certain parts of our prompt set. Particularly we observed that the OpenAI 4o model proactively refused to generate certain images:
🚫 Styles of living artists: completely blocked
🚫 Copyrighted characters (e.g., Darth Vader, Pokémon): initially generated but subsequently blocked
Overall, OpenAI 4o stands out significantly in alignment and coherence, especially excelling in certain unusual prompts that have historically caused issues such as: 'A chair on a cat.' See the images for more examples!
- 1 reply
reacted to
jasoncorkill's
post with 👍
21 days ago
Post
3798
At Rapidata, we compared DeepL with LLMs like DeepSeek-R1, Llama, and Mixtral for translation quality using feedback from over 51,000 native speakers. Despite the costs, the performance makes it a valuable investment, especially in critical applications where translation quality is paramount. Now we can say that Europe is more than imposing regulations.
Our dataset, based on these comparisons, is now available on Hugging Face. This might be useful for anyone working on AI translation or language model evaluation.
Rapidata/Translation-deepseek-llama-mixtral-v-deepl
Our dataset, based on these comparisons, is now available on Hugging Face. This might be useful for anyone working on AI translation or language model evaluation.
Rapidata/Translation-deepseek-llama-mixtral-v-deepl
- 1 reply
reacted to
jasoncorkill's
post with 👍
27 days ago
Post
2208
Benchmarking Google's Veo2: How Does It Compare?
The results did not meet expectations. Veo2 struggled with style consistency and temporal coherence, falling behind competitors like Runway, Pika, Tencent, and even Alibaba. While the model shows promise, its alignment and quality are not yet there.
Google recently launched Veo2, its latest text-to-video model, through select partners like fal.ai. As part of our ongoing evaluation of state-of-the-art generative video models, we rigorously benchmarked Veo2 against industry leaders.
We generated a large set of Veo2 videos spending hundreds of dollars in the process and systematically evaluated them using our Python-based API for human and automated labeling.
Check out the ranking here: https://www.rapidata.ai/leaderboard/video-models
Rapidata/text-2-video-human-preferences-veo2
The results did not meet expectations. Veo2 struggled with style consistency and temporal coherence, falling behind competitors like Runway, Pika, Tencent, and even Alibaba. While the model shows promise, its alignment and quality are not yet there.
Google recently launched Veo2, its latest text-to-video model, through select partners like fal.ai. As part of our ongoing evaluation of state-of-the-art generative video models, we rigorously benchmarked Veo2 against industry leaders.
We generated a large set of Veo2 videos spending hundreds of dollars in the process and systematically evaluated them using our Python-based API for human and automated labeling.
Check out the ranking here: https://www.rapidata.ai/leaderboard/video-models
Rapidata/text-2-video-human-preferences-veo2
liked
2
datasets
28 days ago
liked
2
datasets
about 1 month ago
reacted to
jasoncorkill's
post with 😎
about 2 months ago
Post
2855
Integrating human feedback is vital for evolving AI models. Boost quality, scalability, and cost-effectiveness with our crowdsourcing tool!
..Or run A/B tests and gather thousands of responses in minutes. Upload two images, ask a question, and watch the insights roll in!
Check it out here and let us know your feedback: https://app.rapidata.ai/compare
..Or run A/B tests and gather thousands of responses in minutes. Upload two images, ask a question, and watch the insights roll in!
Check it out here and let us know your feedback: https://app.rapidata.ai/compare
reacted to
jasoncorkill's
post with ❤️🔥
about 2 months ago
Post
4636
Runway Gen-3 Alpha: The Style and Coherence Champion
Runway's latest video generation model, Gen-3 Alpha, is something special. It ranks #3 overall on our text-to-video human preference benchmark, but in terms of style and coherence, it outperforms even OpenAI Sora.
However, it struggles with alignment, making it less predictable for controlled outputs.
We've released a new dataset with human evaluations of Runway Gen-3 Alpha: Rapidata's text-2-video human preferences dataset. If you're working on video generation and want to see how your model compares to the biggest players, we can benchmark it for you.
🚀 DM us if you’re interested!
Dataset: Rapidata/text-2-video-human-preferences-runway-alpha
Runway's latest video generation model, Gen-3 Alpha, is something special. It ranks #3 overall on our text-to-video human preference benchmark, but in terms of style and coherence, it outperforms even OpenAI Sora.
However, it struggles with alignment, making it less predictable for controlled outputs.
We've released a new dataset with human evaluations of Runway Gen-3 Alpha: Rapidata's text-2-video human preferences dataset. If you're working on video generation and want to see how your model compares to the biggest players, we can benchmark it for you.
🚀 DM us if you’re interested!
Dataset: Rapidata/text-2-video-human-preferences-runway-alpha
- 1 reply
liked
2
datasets
about 2 months ago