in preferences that are factually incorrect.from langchain.evaluation import load_evaluatorevaluator = load_evaluator("pairwise_string")evaluator.evaluate_string_pairs(    prediction="Addition is a mathematical operation.",    prediction_b="Addition is a mathematical operation that adds two numbers to create a third number, the 'sum'.",    input="What is addition?",)    {'reasoning': 'Both responses are correct and relevant to the question. However, Response B is more helpful and insightful as it provides a more detailed explanation of what addition is. Response A is correct but lacks depth as it does not explain what the operation of addition entails. \n\nFinal Decision: [[B]]',     'value': 'B',     'score': 0}Defining the Criteria‚ÄãBy default, the LLM is instructed to select the 'preferred' response based on helpfulness, relevance, correctness, and depth of thought. You can customize the criteria by passing in a criteria argument, where the criteria could take any of the following forms:Criteria enum or its string value - to use one of the default criteria and their descriptionsConstitutional principal - use one any of the constitutional principles defined in langchainDictionary: a list of custom criteria, where the key is the name of the criteria, and the value is the description.A list of criteria or constitutional principles - to combine multiple criteria in one.Below is an example for determining preferred writing responses based on a custom style.custom_criteria = {    "simplicity": "Is the language straightforward and unpretentious?",    "clarity": "Are the sentences clear and easy to understand?",    "precision": "Is the writing precise, with no unnecessary words or details?",    "truthfulness": "Does the writing feel honest and sincere?",    "subtext": "Does the writing suggest deeper meanings or themes?",}evaluator = load_evaluator("pairwise_string", criteria=custom_criteria)evaluator.evaluate_string_pairs(    prediction="Every cheerful household
Open In Colab
Open In Colab ->: in preferences that are factually incorrect.from langchain.evaluation import load_evaluatorevaluator = load_evaluator("pairwise_string")evaluator.evaluate_string_pairs(    prediction="Addition is a mathematical operation.",    prediction_b="Addition is a mathematical operation that adds two numbers to create a third number, the 'sum'.",    input="What is addition?",)    {'reasoning': 'Both responses are correct and relevant to the question. However, Response B is more helpful and insightful as it provides a more detailed explanation of what addition is. Response A is correct but lacks depth as it does not explain what the operation of addition entails. \n\nFinal Decision: [[B]]',     'value': 'B',     'score': 0}Defining the Criteria‚ÄãBy default, the LLM is instructed to select the 'preferred' response based on helpfulness, relevance, correctness, and depth of thought. You can customize the criteria by passing in a criteria argument, where the criteria could take any of the following forms:Criteria enum or its string value - to use one of the default criteria and their descriptionsConstitutional principal - use one any of the constitutional principles defined in langchainDictionary: a list of custom criteria, where the key is the name of the criteria, and the value is the description.A list of criteria or constitutional principles - to combine multiple criteria in one.Below is an example for determining preferred writing responses based on a custom style.custom_criteria = {    "simplicity": "Is the language straightforward and unpretentious?",    "clarity": "Are the sentences clear and easy to understand?",    "precision": "Is the writing precise, with no unnecessary words or details?",    "truthfulness": "Does the writing feel honest and sincere?",    "subtext": "Does the writing suggest deeper meanings or themes?",}evaluator = load_evaluator("pairwise_string", criteria=custom_criteria)evaluator.evaluate_string_pairs(    prediction="Every cheerful household