provide a full rubric of what you're looking to grade. Below is an example using accuracy.accuracy_criteria = {    "accuracy": """Score 1: The answer is completely unrelated to the reference.Score 3: The answer has minor relevance but does not align with the reference.Score 5: The answer has moderate relevance but contains inaccuracies.Score 7: The answer aligns with the reference but has minor errors or omissions.Score 10: The answer is completely accurate and aligns perfectly with the reference."""}evaluator = load_evaluator(    "labeled_score_string",     criteria=accuracy_criteria,     llm=ChatOpenAI(model="gpt-4"),)# Correcteval_result = evaluator.evaluate_strings(    prediction="You can find them in the dresser's third drawer.",    reference="The socks are in the third drawer in the dresser",    input="Where are my socks?")print(eval_result)    {'reasoning': "The assistant's answer is accurate and aligns perfectly with the reference. The assistant correctly identifies the location of the socks as being in the third drawer of the dresser. Rating: [[10]]", 'score': 10}# Correct but lacking informationeval_result = evaluator.evaluate_strings(    prediction="You can find them in the dresser.",    reference="The socks are in the third drawer in the dresser",    input="Where are my socks?")print(eval_result)    {'reasoning': "The assistant's response is somewhat relevant to the user's query but lacks specific details. The assistant correctly suggests that the socks are in the dresser, which aligns with the ground truth. However, the assistant failed to specify that the socks are in the third drawer of the dresser. This omission could lead to confusion for the user. Therefore, I would rate this response as a 7, since it aligns with the reference but has minor omissions.\n\nRating: [[7]]", 'score': 7}# Incorrecteval_result = evaluator.evaluate_strings(    prediction="You can find them in the dog's bed.",    reference="The socks are in the third drawer in the
The Scoring Evaluator instructs a language model to assess your model's predictions on a specified scale (default is 1-10) based on your custom criteria or rubric. This feature provides a nuanced evaluation instead of a simplistic binary score, aiding in evaluating models against tailored rubrics and comparing model performance on specific tasks.
The Scoring Evaluator instructs a language model to assess your model's predictions on a specified scale (default is 1-10) based on your custom criteria or rubric. This feature provides a nuanced evaluation instead of a simplistic binary score, aiding in evaluating models against tailored rubrics and comparing model performance on specific tasks. ->: provide a full rubric of what you're looking to grade. Below is an example using accuracy.accuracy_criteria = {    "accuracy": """Score 1: The answer is completely unrelated to the reference.Score 3: The answer has minor relevance but does not align with the reference.Score 5: The answer has moderate relevance but contains inaccuracies.Score 7: The answer aligns with the reference but has minor errors or omissions.Score 10: The answer is completely accurate and aligns perfectly with the reference."""}evaluator = load_evaluator(    "labeled_score_string",     criteria=accuracy_criteria,     llm=ChatOpenAI(model="gpt-4"),)# Correcteval_result = evaluator.evaluate_strings(    prediction="You can find them in the dresser's third drawer.",    reference="The socks are in the third drawer in the dresser",    input="Where are my socks?")print(eval_result)    {'reasoning': "The assistant's answer is accurate and aligns perfectly with the reference. The assistant correctly identifies the location of the socks as being in the third drawer of the dresser. Rating: [[10]]", 'score': 10}# Correct but lacking informationeval_result = evaluator.evaluate_strings(    prediction="You can find them in the dresser.",    reference="The socks are in the third drawer in the dresser",    input="Where are my socks?")print(eval_result)    {'reasoning': "The assistant's response is somewhat relevant to the user's query but lacks specific details. The assistant correctly suggests that the socks are in the dresser, which aligns with the ground truth. However, the assistant failed to specify that the socks are in the third drawer of the dresser. This omission could lead to confusion for the user. Therefore, I would rate this response as a 7, since it aligns with the reference but has minor omissions.\n\nRating: [[7]]", 'score': 7}# Incorrecteval_result = evaluator.evaluate_strings(    prediction="You can find them in the dog's bed.",    reference="The socks are in the third drawer in the