Shocking: 2/3 of LLMs fail at 2K context length
text
Shocking: 2/3 of LLMs fail at 2K context length
new_line
new_line
code_your_own_ai makes a great vlog about mostly LLM related AI content. 
text
code_your_own_ai makes a great vlog about mostly LLM related AI content. 
new_line
As I watched the video below, I wondered about current best practices on LLM evaluation. We have benchmarks, we have sota LLMs evaluating LLMs, we have tools evaluating based on human comparison. 
text
As I watched the video below, I wondered about current best practices on LLM evaluation. We have benchmarks, we have sota LLMs evaluating LLMs, we have tools evaluating based on human comparison. 
new_line
Often, I hear, just play with the LLM for 15 mins to form an opinion. 
text
Often, I hear, just play with the LLM for 15 mins to form an opinion. 
new_line
While I think for a specific use case and clear expectations, this could yield signal carrying experiences, I also see that one prompt is used to judge models. 
text
While I think for a specific use case and clear expectations, this could yield signal carrying experiences, I also see that one prompt is used to judge models. 
new_line
While benchmarks have their weaknesses, and are by themselves not enough to judge model quality, I still think systematic methods that try to reduce various scientifically known errs should be the way forward, even for qualitative estimates. 
text
While benchmarks have their weaknesses, and are by themselves not enough to judge model quality, I still think systematic methods that try to reduce various scientifically known errs should be the way forward, even for qualitative estimates. 
new_line
What do you think? How can we make a public tool for judging models like lmsys/chatbot-arena-leaderboard help to leverage standards known in social science? 
text
What do you think? How can we make a public tool for judging models like lmsys/chatbot-arena-leaderboard help to leverage standards known in social science? 
new_line
new_line
https://www.youtube.com/watch?v=mWrivekFZMM
https://www.youtube.com/watch?v=mWrivekFZMM
link
Shocking: 2/3 of LLMs fail at 2K context length

code_your_own_ai makes a great vlog about mostly LLM related AI content. 
As I watched the video below, I wondered about current best practices on LLM evaluation. We have benchmarks, we have sota LLMs evaluating LLMs, we have tools evaluating based on human comparison. 
Often, I hear, just play with the LLM for 15 mins to form an opinion. 
While I think for a specific use case and clear expectations, this could yield signal carrying experiences, I also see that one prompt is used to judge models. 
While benchmarks have their weaknesses, and are by themselves not enough to judge model quality, I still think systematic methods that try to reduce various scientifically known errs should be the way forward, even for qualitative estimates. 
What do you think? How can we make a public tool for judging models like lmsys/chatbot-arena-leaderboard help to leverage standards known in social science? 

https://www.youtube.com/watch?v=mWrivekFZMM
🤯
clem osanseviero julien-c pierrci NameeO radames AdinaY ucalyptus frontenbrecher
👍
samusenps trtm
63c074053bdc86f81088aae2
https://cdn-avatars.huggingface.co/v1/production/uploads/1673563939929-63c074053bdc86f81088aae2.png
Namee Oberst
NameeO
user
63732ebbbd81fae2b3aaf3fb
https://cdn-avatars.huggingface.co/v1/production/uploads/1669551186189-63732ebbbd81fae2b3aaf3fb.jpeg
Knut Jägersberg
KnutJaegersberg
user
630fdd96a119d49bc1e770d5
https://cdn-avatars.huggingface.co/v1/production/uploads/630fdd96a119d49bc1e770d5/OpU95S4a8hkM8OUCZq79R.jpeg
Adam
adamo1139
user
62a3bb1cd0d8c2c2169f0b88
https://cdn-avatars.huggingface.co/v1/production/uploads/62a3bb1cd0d8c2c2169f0b88/eT2TS0IlQbZtz-F_zHLz9.jpeg
Joseph [open/acc] Pollack
Tonic
user