content.raw: Shocking: 2/3 of LLMs fail at 2K context length
content.type: text
content.value: Shocking: 2/3 of LLMs fail at 2K context length
content.type: new_line
content.type: new_line
content.raw: code_your_own_ai makes a great vlog about mostly LLM related AI content. 
content.type: text
content.value: code_your_own_ai makes a great vlog about mostly LLM related AI content. 
content.type: new_line
content.raw: As I watched the video below, I wondered about current best practices on LLM evaluation. We have benchmarks, we have sota LLMs evaluating LLMs, we have tools evaluating based on human comparison. 
content.type: text
content.value: As I watched the video below, I wondered about current best practices on LLM evaluation. We have benchmarks, we have sota LLMs evaluating LLMs, we have tools evaluating based on human comparison. 
content.type: new_line
content.raw: Often, I hear, just play with the LLM for 15 mins to form an opinion. 
content.type: text
content.value: Often, I hear, just play with the LLM for 15 mins to form an opinion. 
content.type: new_line
content.raw: While I think for a specific use case and clear expectations, this could yield signal carrying experiences, I also see that one prompt is used to judge models. 
content.type: text
content.value: While I think for a specific use case and clear expectations, this could yield signal carrying experiences, I also see that one prompt is used to judge models. 
content.type: new_line
content.raw: While benchmarks have their weaknesses, and are by themselves not enough to judge model quality, I still think systematic methods that try to reduce various scientifically known errs should be the way forward, even for qualitative estimates. 
content.type: text
content.value: While benchmarks have their weaknesses, and are by themselves not enough to judge model quality, I still think systematic methods that try to reduce various scientifically known errs should be the way forward, even for qualitative estimates. 
content.type: new_line
content.raw: What do you think? How can we make a public tool for judging models like lmsys/chatbot-arena-leaderboard help to leverage standards known in social science? 
content.type: text
content.value: What do you think? How can we make a public tool for judging models like lmsys/chatbot-arena-leaderboard help to leverage standards known in social science? 
content.type: new_line
content.type: new_line
content.href: https://www.youtube.com/watch?v=mWrivekFZMM
content.raw: https://www.youtube.com/watch?v=mWrivekFZMM
content.type: link
rawContent: Shocking: 2/3 of LLMs fail at 2K context length
  
  code_your_own_ai makes a great vlog about mostly LLM related AI content. 
  As I watched the video below, I wondered about current best practices on LLM evaluation. We have benchmarks, we have sota LLMs evaluating LLMs, we have tools evaluating based on human comparison. 
  Often, I hear, just play with the LLM for 15 mins to form an opinion. 
  While I think for a specific use case and clear expectations, this could yield signal carrying experiences, I also see that one prompt is used to judge models. 
  While benchmarks have their weaknesses, and are by themselves not enough to judge model quality, I still think systematic methods that try to reduce various scientifically known errs should be the way forward, even for qualitative estimates. 
  What do you think? How can we make a public tool for judging models like lmsys/chatbot-arena-leaderboard help to leverage standards known in social science? 
  
  https://www.youtube.com/watch?v=mWrivekFZMM
reactions.reaction: 🤯
reactions.users: clem osanseviero julien-c pierrci NameeO radames AdinaY ucalyptus frontenbrecher
reactions.reaction: 👍
reactions.users: samusenps trtm
commentators._id: 63c074053bdc86f81088aae2
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673563939929-63c074053bdc86f81088aae2.png
commentators.fullname: Namee Oberst
commentators.name: NameeO
commentators.type: user
commentators._id: 63732ebbbd81fae2b3aaf3fb
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669551186189-63732ebbbd81fae2b3aaf3fb.jpeg
commentators.fullname: Knut Jägersberg
commentators.name: KnutJaegersberg
commentators.type: user
commentators._id: 630fdd96a119d49bc1e770d5
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630fdd96a119d49bc1e770d5/OpU95S4a8hkM8OUCZq79R.jpeg
commentators.fullname: Adam
commentators.name: adamo1139
commentators.type: user
commentators._id: 62a3bb1cd0d8c2c2169f0b88
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a3bb1cd0d8c2c2169f0b88/eT2TS0IlQbZtz-F_zHLz9.jpeg
commentators.fullname: Joseph [open/acc] Pollack
commentators.name: Tonic
commentators.type: user