Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesComparing Chain OutputsFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesEvaluationExamplesComparing Chain OutputsOn this pageComparing Chain OutputsSuppose you have two different prompts (or LLMs). How do you know which will generate "better" results?One automated way to predict the preferred configuration is to use a PairwiseStringEvaluator like the PairwiseStringEvalChain[1]. This chain prompts an LLM to select which output is preferred, given a specific input.For this evaluation, we will need 3 things:An evaluatorA dataset of inputs2 (or more) LLMs, Chains, or Agents to compareThen we will aggregate the results to determine the preferred model.Step 1. Create the Evaluator‚ÄãIn this example, you will use gpt-4 to select which output is preferred.from langchain.evaluation import load_evaluatoreval_chain = load_evaluator("pairwise_string")Step 2. Select Dataset‚ÄãIf you already have real usage data for your LLM, you can use a representative sample. More examples
Open In Colab
Open In Colab ->: Skip to main contentü¶úÔ∏èüîó LangChainDocsUse casesIntegrationsAPICommunityChat our docsLangSmithJS/TS DocsSearchCTRLKGet startedIntroductionInstallationQuickstartLangChain Expression LanguageInterfaceHow toCookbookLangChain Expression Language (LCEL)Why use LCEL?ModulesModel I/‚ÄãORetrievalChainsMemoryAgentsCallbacksModulesSecurityGuidesAdaptersDebuggingDeploymentEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesComparing Chain OutputsFallbacksLangSmithRun LLMs locallyModel comparisonPrivacyPydantic compatibilitySafetyMoreGuidesEvaluationExamplesComparing Chain OutputsOn this pageComparing Chain OutputsSuppose you have two different prompts (or LLMs). How do you know which will generate "better" results?One automated way to predict the preferred configuration is to use a PairwiseStringEvaluator like the PairwiseStringEvalChain[1]. This chain prompts an LLM to select which output is preferred, given a specific input.For this evaluation, we will need 3 things:An evaluatorA dataset of inputs2 (or more) LLMs, Chains, or Agents to compareThen we will aggregate the results to determine the preferred model.Step 1. Create the Evaluator‚ÄãIn this example, you will use gpt-4 to select which output is preferred.from langchain.evaluation import load_evaluatoreval_chain = load_evaluator("pairwise_string")Step 2. Select Dataset‚ÄãIf you already have real usage data for your LLM, you can use a representative sample. More examples