custom evaluators, please refer to the LangSmith documentation.from langchain.evaluation import EvaluatorTypefrom langchain.smith import RunEvalConfigevaluation_config = RunEvalConfig(    # Evaluators can either be an evaluator type (e.g., "qa", "criteria", "embedding_distance", etc.) or a configuration for that evaluator    evaluators=[        # Measures whether a QA response is "Correct", based on a reference answer        # You can also select via the raw string "qa"        EvaluatorType.QA,        # Measure the embedding distance between the output and the reference answer        # Equivalent to: EvalConfig.EmbeddingDistance(embeddings=OpenAIEmbeddings())        EvaluatorType.EMBEDDING_DISTANCE,        # Grade whether the output satisfies the stated criteria.        # You can select a default one such as "helpfulness" or provide your own.        RunEvalConfig.LabeledCriteria("helpfulness"),        # The LabeledScoreString evaluator outputs a score on a scale from 1-10.        # You can use default criteria or write our own rubric        RunEvalConfig.LabeledScoreString(            {                 "accuracy": """Score 1: The answer is completely unrelated to the reference.Score 3: The answer has minor relevance but does not align with the reference.Score 5: The answer has moderate relevance but contains inaccuracies.Score 7: The answer aligns with the reference but has minor errors or omissions.Score 10: The answer is completely accurate and aligns perfectly with the reference."""            },            normalize_by=10,        ),    ],    # You can add custom StringEvaluator or RunEvaluator objects here as well, which will automatically be    # applied to each prediction. Check out the docs for examples.    custom_evaluators=[],)4. Run the agent and evaluators‚ÄãUse the run_on_dataset (or asynchronous arun_on_dataset) function to evaluate your model. This will:Fetch example rows from the specified dataset.Run your agent (or any custom function) on each
Open In Colab
Open In Colab ->: custom evaluators, please refer to the LangSmith documentation.from langchain.evaluation import EvaluatorTypefrom langchain.smith import RunEvalConfigevaluation_config = RunEvalConfig(    # Evaluators can either be an evaluator type (e.g., "qa", "criteria", "embedding_distance", etc.) or a configuration for that evaluator    evaluators=[        # Measures whether a QA response is "Correct", based on a reference answer        # You can also select via the raw string "qa"        EvaluatorType.QA,        # Measure the embedding distance between the output and the reference answer        # Equivalent to: EvalConfig.EmbeddingDistance(embeddings=OpenAIEmbeddings())        EvaluatorType.EMBEDDING_DISTANCE,        # Grade whether the output satisfies the stated criteria.        # You can select a default one such as "helpfulness" or provide your own.        RunEvalConfig.LabeledCriteria("helpfulness"),        # The LabeledScoreString evaluator outputs a score on a scale from 1-10.        # You can use default criteria or write our own rubric        RunEvalConfig.LabeledScoreString(            {                 "accuracy": """Score 1: The answer is completely unrelated to the reference.Score 3: The answer has minor relevance but does not align with the reference.Score 5: The answer has moderate relevance but contains inaccuracies.Score 7: The answer aligns with the reference but has minor errors or omissions.Score 10: The answer is completely accurate and aligns perfectly with the reference."""            },            normalize_by=10,        ),    ],    # You can add custom StringEvaluator or RunEvaluator objects here as well, which will automatically be    # applied to each prediction. Check out the docs for examples.    custom_evaluators=[],)4. Run the agent and evaluators‚ÄãUse the run_on_dataset (or asynchronous arun_on_dataset) function to evaluate your model. This will:Fetch example rows from the specified dataset.Run your agent (or any custom function) on each