Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationString EvaluatorsComparison EvaluatorsPairwise string comparisonPairwise embedding distanceCustom pairwise evaluatorTrajectory EvaluatorsExamplesFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesEvaluationComparison EvaluatorsPairwise string comparisonOn this pagePairwise string comparisonOften you will want to compare predictions of an LLM, Chain, or Agent for a given input. The StringComparison evaluators facilitate this so you can answer questions like:Which LLM or prompt produces a preferred output for a given question?Which examples should I include for few-shot example selection?Which output is better to include for fine-tuning?The simplest and often most reliable automated way to choose a preferred prediction for a given input is to use the pairwise_string evaluator.Check out the reference docs for the PairwiseStringEvalChain for more info.from langchain.evaluation import load_evaluatorevaluator = load_evaluator("labeled_pairwise_string")evaluator.evaluate_string_pairs(    prediction="there are three dogs",    prediction_b="4",    input="how many dogs are in the park?",    reference="four",)    {'reasoning': 'Both responses are relevant to the question asked, as they both provide a numerical answer to the question about the number of dogs in the park. However, Response A is incorrect according to the reference answer, which states that there are four dogs. Response B, on the other hand, is correct as it matches the reference answer. Neither response demonstrates depth of thought, as they both simply provide a numerical answer without any
Open In Colab
Open In Colab ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationString EvaluatorsComparison EvaluatorsPairwise string comparisonPairwise embedding distanceCustom pairwise evaluatorTrajectory EvaluatorsExamplesFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesEvaluationComparison EvaluatorsPairwise string comparisonOn this pagePairwise string comparisonOften you will want to compare predictions of an LLM, Chain, or Agent for a given input. The StringComparison evaluators facilitate this so you can answer questions like:Which LLM or prompt produces a preferred output for a given question?Which examples should I include for few-shot example selection?Which output is better to include for fine-tuning?The simplest and often most reliable automated way to choose a preferred prediction for a given input is to use the pairwise_string evaluator.Check out the reference docs for the PairwiseStringEvalChain for more info.from langchain.evaluation import load_evaluatorevaluator = load_evaluator("labeled_pairwise_string")evaluator.evaluate_string_pairs(    prediction="there are three dogs",    prediction_b="4",    input="how many dogs are in the park?",    reference="four",)    {'reasoning': 'Both responses are relevant to the question asked, as they both provide a numerical answer to the question about the number of dogs in the park. However, Response A is incorrect according to the reference answer, which states that there are four dogs. Response B, on the other hand, is correct as it matches the reference answer. Neither response demonstrates depth of thought, as they both simply provide a numerical answer without any