content.raw: It feels awkward having my first post sharing my stuff, but this is a weekend project that I really enjoyed working on. I'd love to meet more people interested in random ideas like this.
content.type: text
content.value: It feels awkward having my first post sharing my stuff, but this is a weekend project that I really enjoyed working on. I'd love to meet more people interested in random ideas like this.
content.type: new_line
content.type: new_line
content.raw: A hard part of building AI applications is choosing which model to use. What if we don’t have to? What if we can predict the best model for any prompt?
content.type: text
content.value: A hard part of building AI applications is choosing which model to use. What if we don’t have to? What if we can predict the best model for any prompt?
content.type: new_line
content.type: new_line
content.raw: Predictive human preference aims to predict which model users might prefer for a specific query.
content.type: text
content.value: Predictive human preference aims to predict which model users might prefer for a specific query.
content.type: new_line
content.type: new_line
content.href: https://huyenchip.com/2024/02/28/predictive-human-preference.html
content.raw: https://huyenchip.com/2024/02/28/predictive-human-preference.html
content.type: link
content.type: new_line
content.type: new_line
content.raw: One use case is model routing. If we know in advance that for a prompt, users will prefer Claude Instant’s response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant. Model routing has the potential to increase response quality while reducing costs and latency.
content.type: text
content.value: One use case is model routing. If we know in advance that for a prompt, users will prefer Claude Instant’s response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant. Model routing has the potential to increase response quality while reducing costs and latency.
content.type: new_line
content.type: new_line
content.raw: One pattern is that for simple prompts, weak models can do (nearly) as well as strong models. For more challenging prompts, however, users are more likely to prefer stronger models. Here’s a visualization of predicted human preference for an easy prompt (“hello, how are you?”) and a challenging prompt (“Explain why Planc length …”).
content.type: text
content.value: One pattern is that for simple prompts, weak models can do (nearly) as well as strong models. For more challenging prompts, however, users are more likely to prefer stronger models. Here’s a visualization of predicted human preference for an easy prompt (“hello, how are you?”) and a challenging prompt (“Explain why Planc length …”).
content.type: new_line
content.type: new_line
content.raw: Preference predictors make it possible to create leaderboards unique to any prompt and domain.
content.type: text
content.value: Preference predictors make it possible to create leaderboards unique to any prompt and domain.
rawContent: It feels awkward having my first post sharing my stuff, but this is a weekend project that I really enjoyed working on. I'd love to meet more people interested in random ideas like this.
  
  A hard part of building AI applications is choosing which model to use. What if we don’t have to? What if we can predict the best model for any prompt?
  
  Predictive human preference aims to predict which model users might prefer for a specific query.
  
  https://huyenchip.com/2024/02/28/predictive-human-preference.html
  
  One use case is model routing. If we know in advance that for a prompt, users will prefer Claude Instant’s response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant. Model routing has the potential to increase response quality while reducing costs and latency.
  
  One pattern is that for simple prompts, weak models can do (nearly) as well as strong models. For more challenging prompts, however, users are more likely to prefer stronger models. Here’s a visualization of predicted human preference for an easy prompt (“hello, how are you?”) and a challenging prompt (“Explain why Planc length …”).
  
  Preference predictors make it possible to create leaderboards unique to any prompt and domain.
reactions.reaction: ❤️
reactions.users: clem Nganx andrewrreed vladbogo osanseviero loubnabnl EmilyWitko ankity09 samusenps callmesan bedestaz ChavyvAkvar dortorwu2 gsarti Shaheer-ipynb aotrih ntdas dark-pen julien-c kgourgou pcuenq valeriiakuka sbrandeis
reactions.reaction: 🤗
reactions.users: andrewrreed mvaloatto kramp julien-c kgourgou gigant Bkarine
reactions.reaction: 👍
reactions.users: FremyCompany derek-thomas ron-wolf
commentators._id: 5dd96eb166059660ed1ee413
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/NQtzmrDdbG0H8qkZvRyGk.jpeg
commentators.fullname: Julien Chaumond
commentators.name: julien-c
commentators.type: user
commentators._id: 61d375fd733d3a83ecd1bba9
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61d375fd733d3a83ecd1bba9/oIXwvvs1-HaCnJXMCZgkc.jpeg
commentators.fullname: Andrew Reed
commentators.name: andrewrreed
commentators.type: user
commentators._id: 6032802e1f993496bc14d9e3
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6032802e1f993496bc14d9e3/w6hr-DEQot4VVkoyRIBiy.png
commentators.fullname: Omar Sanseviero
commentators.name: osanseviero
commentators.type: user