Evaluating Audio Reasoning with Big Bench Audio
•
21
AI & ML interests
None defined yet.
Recent Activity
View all activity
Articles
georgewritescode
posted
an
update
3 days ago
Post
969
🎵 Announcing Artificial Analysis Music Arena! Vote for songs generated by leading music models across genres from pop to metal to rock & more
Key details:
🏁 Participate in Music Arena and after a day of voting we’ll unveil the world’s first public ranking of AI music models.
✨ Currently featuring models from Suno, Riffusion, Meta, Google, udio and Stability AI!
🎤 Support for both a vocals mode and an instrumental mode
🎸 A diverse array of prompts from genres including pop, RnB, metal, rock, classical, jazz, and more
Check it out here:
ArtificialAnalysis/Music-Arena-Leaderboard
Key details:
🏁 Participate in Music Arena and after a day of voting we’ll unveil the world’s first public ranking of AI music models.
✨ Currently featuring models from Suno, Riffusion, Meta, Google, udio and Stability AI!
🎤 Support for both a vocals mode and an instrumental mode
🎸 A diverse array of prompts from genres including pop, RnB, metal, rock, classical, jazz, and more
Check it out here:
ArtificialAnalysis/Music-Arena-Leaderboard
georgewritescode
updated
a
Space
4 days ago
georgewritescode
published
a
Space
4 days ago
georgewritescode
updated
3
Spaces
about 1 month ago
georgewritescode
published
a
Space
about 1 month ago
mhillsmith
updated
2
datasets
7 months ago
georgewritescode
updated
a
dataset
7 months ago
When include new models like SORA and HunYuan?
3
#2 opened 8 months ago
by
RaphaelLiu
georgewritescode
updated
a
Space
about 1 year ago
georgewritescode
posted
an
update
about 1 year ago
Post
1112
Visualization of GPT-4o breaking away from the quality & speed trade-off curve the LLMs have followed thus far ✂️
Key GPT-4o takeaways
‣ GPT-4o not only offers the highest quality, it also sits amongst the fastest LLMs
‣ For those with speed/latency-sensitive use cases, where previously Claude 3 Haiku or Mixtral 8x7b were leaders, GPT-4o is now a compelling option (though significantly more expensive)
‣ Previously Groq was the only provider to break from the curve using its own LPU chips. OpenAI has done it on Nvidia hardware (one can imagine the potential for GPT-4o on Groq)
👉 How did they do it? Will follow up with more analysis on this but potential approaches include a very large but sparse MoE model (similar to Snowflake's Arctic) and improvements in data quality (likely to have driven much of Llama 3's impressive quality relative to parameter count)
Notes: Throughput represents the median across providers over the last 14 days of measurements (8x per day)
Data is present on our HF leaderboard: ArtificialAnalysis/LLM-Performance-Leaderboard and graphs present on our website
Key GPT-4o takeaways
‣ GPT-4o not only offers the highest quality, it also sits amongst the fastest LLMs
‣ For those with speed/latency-sensitive use cases, where previously Claude 3 Haiku or Mixtral 8x7b were leaders, GPT-4o is now a compelling option (though significantly more expensive)
‣ Previously Groq was the only provider to break from the curve using its own LPU chips. OpenAI has done it on Nvidia hardware (one can imagine the potential for GPT-4o on Groq)
👉 How did they do it? Will follow up with more analysis on this but potential approaches include a very large but sparse MoE model (similar to Snowflake's Arctic) and improvements in data quality (likely to have driven much of Llama 3's impressive quality relative to parameter count)
Notes: Throughput represents the median across providers over the last 14 days of measurements (8x per day)
Data is present on our HF leaderboard: ArtificialAnalysis/LLM-Performance-Leaderboard and graphs present on our website
- 1 reply
georgewritescode
posted
an
update
about 1 year ago
Post
2368
Excited to bring our benchmarking leaderboard of >100 LLM API endpoints to HF!
Speed and price are often just as important as quality when building applications with LLMs. We bring together all the data you need to consider all three when you need to pick a model and API provider.
Coverage:
‣ Quality (Index of evals, MMLU, Chatbot Arena, HumanEval, MT-Bench)
‣ Throughput (tokens/s: median, P5, P25, P75, P95)
‣ Latency (TTFT: median, P5, P25, P75, P95)
‣ Context window
‣ OpenAI library compatibility
Link to Space: ArtificialAnalysis/LLM-Performance-Leaderboard
Blog post: https://huggingface.co/blog/leaderboard-artificial-analysis
Speed and price are often just as important as quality when building applications with LLMs. We bring together all the data you need to consider all three when you need to pick a model and API provider.
Coverage:
‣ Quality (Index of evals, MMLU, Chatbot Arena, HumanEval, MT-Bench)
‣ Throughput (tokens/s: median, P5, P25, P75, P95)
‣ Latency (TTFT: median, P5, P25, P75, P95)
‣ Context window
‣ OpenAI library compatibility
Link to Space: ArtificialAnalysis/LLM-Performance-Leaderboard
Blog post: https://huggingface.co/blog/leaderboard-artificial-analysis