of thought?'} template='Given the input context, which do you prefer: A or B?\nEvaluate based on the following criteria:\n{criteria}\nReason step by step and finally, respond with either [[A]] or [[B]] on its own line.\n\nDATA\n----\ninput: {input}\nreference: {reference}\nA: {prediction}\nB: {prediction_b}\n---\nReasoning:\n\n' template_format='f-string' validate_template=Trueevaluator.evaluate_string_pairs(    prediction="The dog that ate the ice cream was named fido.",    prediction_b="The dog's name is spot",    input="What is the name of the dog that ate the ice cream?",    reference="The dog's name is fido",)    {'reasoning': 'Helpfulness: Both A and B are helpful as they provide a direct answer to the question.\nRelevance: A is relevant as it refers to the correct name of the dog from the text. B is not relevant as it provides a different name.\nCorrectness: A is correct as it accurately states the name of the dog. B is incorrect as it provides a different name.\nDepth: Both A and B demonstrate a similar level of depth as they both provide a straightforward answer to the question.\n\nGiven these evaluations, the preferred response is:\n',     'value': 'A',     'score': 1}PreviousComparison EvaluatorsNextPairwise embedding distanceMethodsWithout ReferencesDefining the CriteriaCustomize the LLMCustomize the Evaluation PromptCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.
Open In Colab
Open In Colab ->: of thought?'} template='Given the input context, which do you prefer: A or B?\nEvaluate based on the following criteria:\n{criteria}\nReason step by step and finally, respond with either [[A]] or [[B]] on its own line.\n\nDATA\n----\ninput: {input}\nreference: {reference}\nA: {prediction}\nB: {prediction_b}\n---\nReasoning:\n\n' template_format='f-string' validate_template=Trueevaluator.evaluate_string_pairs(    prediction="The dog that ate the ice cream was named fido.",    prediction_b="The dog's name is spot",    input="What is the name of the dog that ate the ice cream?",    reference="The dog's name is fido",)    {'reasoning': 'Helpfulness: Both A and B are helpful as they provide a direct answer to the question.\nRelevance: A is relevant as it refers to the correct name of the dog from the text. B is not relevant as it provides a different name.\nCorrectness: A is correct as it accurately states the name of the dog. B is incorrect as it provides a different name.\nDepth: Both A and B demonstrate a similar level of depth as they both provide a straightforward answer to the question.\n\nGiven these evaluations, the preferred response is:\n',     'value': 'A',     'score': 1}PreviousComparison EvaluatorsNextPairwise embedding distanceMethodsWithout ReferencesDefining the CriteriaCustomize the LLMCustomize the Evaluation PromptCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright ¬© 2023 LangChain, Inc.