2+2?" is indeed "four". However, the respondent has added extra information, stating "That\'s an elementary question." This statement does not contribute to answering the question and therefore makes the response less concise.\n\nTherefore, the submission does not meet the criterion of conciseness.\n\nN', 'value': 'N', 'score': 0}Output Format‚ÄãAll string evaluators expose an evaluate_strings (or async aevaluate_strings) method, which accepts:input (str) ‚Äì The input to the agent.prediction (str) ‚Äì The predicted response.The criteria evaluators return a dictionary with the following values:score: Binary integer 0 to 1, where 1 would mean that the output is compliant with the criteria, and 0 otherwisevalue: A "Y" or "N" corresponding to the scorereasoning: String "chain of thought reasoning" from the LLM generated prior to creating the scoreUsing Reference Labels‚ÄãSome criteria (such as correctness) require reference labels to work correctly. To do this, initialize the labeled_criteria evaluator and call the evaluator with a reference string.evaluator = load_evaluator("labeled_criteria", criteria="correctness")# We can even override the model's learned knowledge using ground truth labelseval_result = evaluator.evaluate_strings(    input="What is the capital of the US?",    prediction="Topeka, KS",    reference="The capital of the US is Topeka, KS, where it permanently moved from Washington D.C. on May 16, 2023",)print(f'With ground truth: {eval_result["score"]}')    With ground truth: 1Default CriteriaMost of the time, you'll want to define your own custom criteria (see below), but we also provide some common criteria you can load with a single string.
Open In Colab
Open In Colab ->: 2+2?" is indeed "four". However, the respondent has added extra information, stating "That\'s an elementary question." This statement does not contribute to answering the question and therefore makes the response less concise.\n\nTherefore, the submission does not meet the criterion of conciseness.\n\nN', 'value': 'N', 'score': 0}Output Format‚ÄãAll string evaluators expose an evaluate_strings (or async aevaluate_strings) method, which accepts:input (str) ‚Äì The input to the agent.prediction (str) ‚Äì The predicted response.The criteria evaluators return a dictionary with the following values:score: Binary integer 0 to 1, where 1 would mean that the output is compliant with the criteria, and 0 otherwisevalue: A "Y" or "N" corresponding to the scorereasoning: String "chain of thought reasoning" from the LLM generated prior to creating the scoreUsing Reference Labels‚ÄãSome criteria (such as correctness) require reference labels to work correctly. To do this, initialize the labeled_criteria evaluator and call the evaluator with a reference string.evaluator = load_evaluator("labeled_criteria", criteria="correctness")# We can even override the model's learned knowledge using ground truth labelseval_result = evaluator.evaluate_strings(    input="What is the capital of the US?",    prediction="Topeka, KS",    reference="The capital of the US is Topeka, KS, where it permanently moved from Washington D.C. on May 16, 2023",)print(f'With ground truth: {eval_result["score"]}')    With ground truth: 1Default CriteriaMost of the time, you'll want to define your own custom criteria (see below), but we also provide some common criteria you can load with a single string.