content.raw: Question about LightEval 🤗:
content.type: text
content.value: Question about LightEval 🤗:
content.type: new_line
content.type: new_line
content.raw: I've been searching for an LLM evaluation suite that can, out-of-the-box, compare the outputs of a model(s) without any enhancements vs. the same model with better prompt engineering, vs. the same model with RAG vs. the same model with fine-tuning.
content.type: text
content.value: I've been searching for an LLM evaluation suite that can, out-of-the-box, compare the outputs of a model(s) without any enhancements vs. the same model with better prompt engineering, vs. the same model with RAG vs. the same model with fine-tuning.
content.type: new_line
content.type: new_line
content.raw: I unfortunately have not found a tool that fits my exact description, but of course I ran into LightEval.
content.type: text
content.value: I unfortunately have not found a tool that fits my exact description, but of course I ran into LightEval.
content.type: new_line
content.type: new_line
content.raw: A huge pain-point of building large-scale projects that use LLMs is that prior to building an MVP, it is difficult to evaluate whether better prompt engineering, or RAG, or fine-tuning, or some combination of all is needed for satisfactory LLM output in terms of the project's given use case. 
content.type: text
content.value: A huge pain-point of building large-scale projects that use LLMs is that prior to building an MVP, it is difficult to evaluate whether better prompt engineering, or RAG, or fine-tuning, or some combination of all is needed for satisfactory LLM output in terms of the project's given use case. 
content.type: new_line
content.type: new_line
content.raw: Time and resources is then wasted R&D'ing exactly what LLM enhancements are needed. 
content.type: text
content.value: Time and resources is then wasted R&D'ing exactly what LLM enhancements are needed. 
content.type: new_line
content.type: new_line
content.raw: I believe an out-of-the-box solution to compare models w/ or w/out the aforementioned LLM enhancements could help teams of any size better decide what LLM enhancements are needed prior to building.
content.type: text
content.value: I believe an out-of-the-box solution to compare models w/ or w/out the aforementioned LLM enhancements could help teams of any size better decide what LLM enhancements are needed prior to building.
content.type: new_line
content.type: new_line
content.raw: I wanted to know if the LightEval team or Hugging Face in general is thinking about such a tool. 
content.type: text
content.value: I wanted to know if the LightEval team or Hugging Face in general is thinking about such a tool. 
rawContent: Question about LightEval 🤗:
  
  I've been searching for an LLM evaluation suite that can, out-of-the-box, compare the outputs of a model(s) without any enhancements vs. the same model with better prompt engineering, vs. the same model with RAG vs. the same model with fine-tuning.
  
  I unfortunately have not found a tool that fits my exact description, but of course I ran into LightEval.
  
  A huge pain-point of building large-scale projects that use LLMs is that prior to building an MVP, it is difficult to evaluate whether better prompt engineering, or RAG, or fine-tuning, or some combination of all is needed for satisfactory LLM output in terms of the project's given use case. 
  
  Time and resources is then wasted R&D'ing exactly what LLM enhancements are needed. 
  
  I believe an out-of-the-box solution to compare models w/ or w/out the aforementioned LLM enhancements could help teams of any size better decide what LLM enhancements are needed prior to building.
  
  I wanted to know if the LightEval team or Hugging Face in general is thinking about such a tool. 
reactions.reaction: 👀
reactions.users: John6666
commentators._id: 63e0eea7af523c37e5a77966
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678663263366-63e0eea7af523c37e5a77966.jpeg
commentators.fullname: Nathan Habib
commentators.name: SaylorTwift
commentators.type: user