Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationString EvaluatorsCriteria EvaluationCustom String EvaluatorEmbedding DistanceExact MatchRegex MatchScoring EvaluatorString DistanceComparison EvaluatorsTrajectory EvaluatorsExamplesFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesEvaluationString EvaluatorsScoring EvaluatorOn this pageScoring EvaluatorThe Scoring Evaluator instructs a language model to assess your model's predictions on a specified scale (default is 1-10) based on your custom criteria or rubric. This feature provides a nuanced evaluation instead of a simplistic binary score, aiding in evaluating models against tailored rubrics and comparing model performance on specific tasks.Before we dive in, please note that any specific grade from an LLM should be taken with a grain of salt. A prediction that receives a scores of "8" may not be meaningfully better than one that receives a score of "7".Usage with Ground Truth‚ÄãFor a thorough understanding, refer to the LabeledScoreStringEvalChain documentation.Below is an example demonstrating the usage of LabeledScoreStringEvalChain using the default prompt:from langchain.evaluation import load_evaluatorfrom langchain.chat_models import ChatOpenAIevaluator = load_evaluator("labeled_score_string", llm=ChatOpenAI(model="gpt-4"))# Correcteval_result = evaluator.evaluate_strings(    prediction="You can find them in the dresser's third drawer.",    reference="The socks are in the third drawer in the dresser",    input="Where are my socks?")print(eval_result)    {'reasoning': "The assistant's response is helpful, accurate, and directly answers the
The Scoring Evaluator instructs a language model to assess your model's predictions on a specified scale (default is 1-10) based on your custom criteria or rubric. This feature provides a nuanced evaluation instead of a simplistic binary score, aiding in evaluating models against tailored rubrics and comparing model performance on specific tasks.
The Scoring Evaluator instructs a language model to assess your model's predictions on a specified scale (default is 1-10) based on your custom criteria or rubric. This feature provides a nuanced evaluation instead of a simplistic binary score, aiding in evaluating models against tailored rubrics and comparing model performance on specific tasks. ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationString EvaluatorsCriteria EvaluationCustom String EvaluatorEmbedding DistanceExact MatchRegex MatchScoring EvaluatorString DistanceComparison EvaluatorsTrajectory EvaluatorsExamplesFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesEvaluationString EvaluatorsScoring EvaluatorOn this pageScoring EvaluatorThe Scoring Evaluator instructs a language model to assess your model's predictions on a specified scale (default is 1-10) based on your custom criteria or rubric. This feature provides a nuanced evaluation instead of a simplistic binary score, aiding in evaluating models against tailored rubrics and comparing model performance on specific tasks.Before we dive in, please note that any specific grade from an LLM should be taken with a grain of salt. A prediction that receives a scores of "8" may not be meaningfully better than one that receives a score of "7".Usage with Ground Truth‚ÄãFor a thorough understanding, refer to the LabeledScoreStringEvalChain documentation.Below is an example demonstrating the usage of LabeledScoreStringEvalChain using the default prompt:from langchain.evaluation import load_evaluatorfrom langchain.chat_models import ChatOpenAIevaluator = load_evaluator("labeled_score_string", llm=ChatOpenAI(model="gpt-4"))# Correcteval_result = evaluator.evaluate_strings(    prediction="You can find them in the dresser's third drawer.",    reference="The socks are in the third drawer in the dresser",    input="Where are my socks?")print(eval_result)    {'reasoning': "The assistant's response is helpful, accurate, and directly answers the