evaluate this so we have a baseline.Evaluate Agent‚ÄãIn addition to logging runs, LangSmith also allows you to test and evaluate your LLM applications.In this section, you will leverage LangSmith to create a benchmark dataset and run AI-assisted evaluators on an agent. You will do so in a few steps:Create a datasetInitialize a new agent to benchmarkConfigure evaluators to grade an agent's outputRun the agent over the dataset and evaluate the results1. Create a LangSmith dataset‚ÄãBelow, we use the LangSmith client to create a dataset from the input questions from above and a list labels. You will use these later to measure performance for a new agent. A dataset is a collection of examples, which are nothing more than input-output pairs you can use as test cases to your application.For more information on datasets, including how to create them from CSVs or other files or how to create them in the platform, please refer to the LangSmith documentation.outputs = [    "LangChain is an open-source framework for building applications using large language models. It is also the name of the company building LangSmith.",    "LangSmith is a unified platform for debugging, testing, and monitoring language model applications and agents powered by LangChain",    "July 18, 2023",    "The langsmith cookbook is a github repository containing detailed examples of how to use LangSmith to debug, evaluate, and monitor large language model-powered applications.",    "September 5, 2023",]dataset_name = f"agent-qa-{unique_id}"dataset = client.create_dataset(    dataset_name, description="An example dataset of questions over the LangSmith documentation.")for query, answer in zip(inputs, outputs):    client.create_example(inputs={"input": query}, outputs={"output": answer}, dataset_id=dataset.id)2. Initialize a new agent to benchmark‚ÄãLangSmith lets you evaluate any LLM, chain, agent, or even a custom function. Conversational agents are stateful (they have memory); to ensure that this
Open In Colab
Open In Colab ->: evaluate this so we have a baseline.Evaluate Agent‚ÄãIn addition to logging runs, LangSmith also allows you to test and evaluate your LLM applications.In this section, you will leverage LangSmith to create a benchmark dataset and run AI-assisted evaluators on an agent. You will do so in a few steps:Create a datasetInitialize a new agent to benchmarkConfigure evaluators to grade an agent's outputRun the agent over the dataset and evaluate the results1. Create a LangSmith dataset‚ÄãBelow, we use the LangSmith client to create a dataset from the input questions from above and a list labels. You will use these later to measure performance for a new agent. A dataset is a collection of examples, which are nothing more than input-output pairs you can use as test cases to your application.For more information on datasets, including how to create them from CSVs or other files or how to create them in the platform, please refer to the LangSmith documentation.outputs = [    "LangChain is an open-source framework for building applications using large language models. It is also the name of the company building LangSmith.",    "LangSmith is a unified platform for debugging, testing, and monitoring language model applications and agents powered by LangChain",    "July 18, 2023",    "The langsmith cookbook is a github repository containing detailed examples of how to use LangSmith to debug, evaluate, and monitor large language model-powered applications.",    "September 5, 2023",]dataset_name = f"agent-qa-{unique_id}"dataset = client.create_dataset(    dataset_name, description="An example dataset of questions over the LangSmith documentation.")for query, answer in zip(inputs, outputs):    client.create_example(inputs={"input": query}, outputs={"output": answer}, dataset_id=dataset.id)2. Initialize a new agent to benchmark‚ÄãLangSmith lets you evaluate any LLM, chain, agent, or even a custom function. Conversational agents are stateful (they have memory); to ensure that this