Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationString EvaluatorsComparison EvaluatorsPairwise string comparisonPairwise embedding distanceCustom pairwise evaluatorTrajectory EvaluatorsExamplesFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesEvaluationComparison EvaluatorsCustom pairwise evaluatorOn this pageCustom pairwise evaluatorYou can make your own pairwise string evaluators by inheriting from PairwiseStringEvaluator class and overwriting the _evaluate_string_pairs method (and the _aevaluate_string_pairs method if you want to use the evaluator asynchronously).In this example, you will make a simple custom evaluator that just returns whether the first prediction has more whitespace tokenized 'words' than the second.You can check out the reference docs for the PairwiseStringEvaluator interface for more info.from typing import Optional, Anyfrom langchain.evaluation import PairwiseStringEvaluatorclass LengthComparisonPairwiseEvaluator(PairwiseStringEvaluator):    """    Custom evaluator to compare two strings.    """    def _evaluate_string_pairs(        self,        *,        prediction: str,        prediction_b: str,        reference: Optional[str] = None,        input: Optional[str] = None,        **kwargs: Any,    ) -> dict:        score = int(len(prediction.split()) > len(prediction_b.split()))        return {"score": score}evaluator = LengthComparisonPairwiseEvaluator()evaluator.evaluate_string_pairs(    prediction="The quick brown fox jumped over the lazy dog.",    prediction_b="The quick brown fox jumped over the dog.",)    {'score': 1}LLM-Based Example‚ÄãThat example was simple to
Open In Colab
Open In Colab ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationString EvaluatorsComparison EvaluatorsPairwise string comparisonPairwise embedding distanceCustom pairwise evaluatorTrajectory EvaluatorsExamplesFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesEvaluationComparison EvaluatorsCustom pairwise evaluatorOn this pageCustom pairwise evaluatorYou can make your own pairwise string evaluators by inheriting from PairwiseStringEvaluator class and overwriting the _evaluate_string_pairs method (and the _aevaluate_string_pairs method if you want to use the evaluator asynchronously).In this example, you will make a simple custom evaluator that just returns whether the first prediction has more whitespace tokenized 'words' than the second.You can check out the reference docs for the PairwiseStringEvaluator interface for more info.from typing import Optional, Anyfrom langchain.evaluation import PairwiseStringEvaluatorclass LengthComparisonPairwiseEvaluator(PairwiseStringEvaluator):    """    Custom evaluator to compare two strings.    """    def _evaluate_string_pairs(        self,        *,        prediction: str,        prediction_b: str,        reference: Optional[str] = None,        input: Optional[str] = None,        **kwargs: Any,    ) -> dict:        score = int(len(prediction.split()) > len(prediction_b.split()))        return {"score": score}evaluator = LengthComparisonPairwiseEvaluator()evaluator.evaluate_string_pairs(    prediction="The quick brown fox jumped over the lazy dog.",    prediction_b="The quick brown fox jumped over the dog.",)    {'score': 1}LLM-Based Example‚ÄãThat example was simple to