Answers.Body: <blockquote>
  <p>Let's imagine that I have an evaluation function <code>f(x)</code> that allow to score a specific result.</p>
  </blockquote>
  <p>It's not clear from this, whether <code>f(x)</code> is a relevence score for a single found item, or is a rating for a complete search result. If it is the former, then you can construct the latter by aggregating results in some semi-arbitrary fashion such as taking an average of <code>f(x)</code> for the top ten results returned by a search. This aggregation is arbitrary, but you can base it on intuitive concepts such as the number of results presented in a page. Ideally you have this grounded in some sense relating to the higher level goals of the system (i.e. although the system is a search system, no-one has &quot;searching&quot; as a stand-alone goal in practice, there is always some context).</p>
  <p>To further aggregate this value per attempt into a value <em>per search agent</em>, you will need to construct a test across multiple requests, covering a gamut of different search terms representative of the kinds of search you expect the system will be used for. Take the mean result or <code>f(x)</code> score over all test searches in the test set. This will be necessary because it is very unlikely that different searches will be equally effective, so you will want some kind of averaged score that represents the system's performance against how it will be used in practice.</p>
  <blockquote>
  <p>For instance, how can I tell between two versions of my agent (v1 and v2), which one is performing better ?</p>
  </blockquote>
  <p>It is the one with the better aggregated <code>f(x)</code> score, measured over a standardised test set. You can reduce cost of measuring this, at the expense of accuracy, by using a smaller fair sample from the test set. Or you could achieve similar by having small/fast test set to get rough indication and then a larger test set for refining things. You will want to understand the standard error in your measures of aggregate <code>f(x)</code> so you can tell when a difference is large enough to be significant.</p>
  <blockquote>
  <p>I don’t have access to the entire database of results due to the sheer volume (millions of results) and the cost involved.</p>
  </blockquote>
  <p>Whilst this might rule out certain kinds of system validation, such as demonstrating the current <code>f(x)</code> is the best possible, it does not prevent you testing the system statistically. The important detail is to make the results on repeated test runs with different agents <em>equivalent</em> in as many ways as possible - changing only the internal and controllable aspects of the agent between tests, not how or what you measure.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-03T09:45:09.380Z
Answers.LastActivityDate: 2024-06-03T18:45:46.087Z
Answers.LastEditDate: 2024-06-03T18:45:46.087Z
Answers.UserDisplayName: Neil Slater
Body: <p>I’m building an AI agent designed to find the best results for with a search tool. The agent is essentially a Large Language Model (LLM) connected to a Search API via function calling. The goal is for this agent to use the Search API to identify the most relevant results.</p>
  <p>Let's imagine that I have an evaluation function <code>f(x)</code> that allow to score a specific result.</p>
  <p>How can I evaluate the performance of my agent ?
  For instance, how can I tell between two versions of my agent (v1 and v2), which one is performing better ?</p>
  <p>Here is a specific constraint due to the fact that in a real world search, I don’t have access to the entire database of results due to the sheer volume (millions of results) and the cost involved. Since I can’t purchase all these results, it’s difficult to know if a specific record should be included in the search results. So I can't use recall like metrics.</p>
  <p>Given this constraint, how can I effectively evaluate the performance of my AI agent?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-30T07:16:10.677Z
Comments.Text: How do you assess leads found by humans doing the same task?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-30T08:04:21.877Z
Comments.Text: Looks like an Unsupervised Learning task to me. So you could _cluster_ your leads by some criterion. There are many criteria, and many methods of cluster analysis.
Comments.UserDisplayName: knb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T08:24:36.970Z
Comments.Text: @NeilSlater that’s a very good question. Typically, this process is done through qualitative human evaluation (in other words, it’s not very scientific or rationalized). But that’s precisely my question! How can I turn this manually handled problem into one that can be solved with AI?
Comments.UserDisplayName: Léonard Henriquez
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T10:29:15.403Z
Comments.Text: That would be useful to know, and helps frame the question as automation of something existing (as opposed to asking for magic, which is close to how the question currently phrases it). Please put some information about how humans performing the same task are assessed into the question. It doesn't have to be detailed, just a variation of your comment. As a spoiler, of course someone answering will be suggesting you assess the AI's outputs in the same way, so you may want to elaborate on the resources you have at your disposal, and what opportunities you have for online feedback.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T10:33:02.953Z
Comments.Text: Another obvious follow-up is do you have data from a history of good and bad leads found by humans to use for training of any assessment function?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T18:09:18.110Z
Comments.Text: @NeilSlater, sorry, my question wasn't clear. I don't need help in finding a business metric to specifically evaluate the quality of a lead. It's more of a generic problem that is "how to evaluate the accuracy of a search done by an AI agent". The lead use case was just to give an example more concrete to the problem. I edited the post to remove mention of the lead use case.
Comments.UserDisplayName: Léonard Henriquez
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T21:22:51.347Z
Comments.Text: OK, so now we're back to "magic" unless you can explain what you want . . . ignoring the AI for a moment, do you have any way to assess search results from your system? Any AI is just going to be a system that tries to maxmise the results from that metric in a more sophisticated manner than a non-AI-based solution. The metric does not need to be _direct_. It could be based on results such as saving the searchers time, making them happy, or some financially-measured benefit. However the more direct and individually targetted metrics are easier to use for training.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T21:24:39.100Z
Comments.Text: Also the metric does not need to be something assessed on all results forever. You could collect and score data about good and bad search results, and use that to train an agent that you may reasonably *expect* to perform at a certain level. So, do you have any way at all to tell whether one search result is better than another, or to give it a rating, assign a financial benefit etc?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-03T07:50:13.187Z
Comments.Text: OK @NeilSlater, there nothing related to magic here. I'm asking for a scientific method to evaluate search results. Evaluating search results is a problem like many other in LLM evaluation. Let's say that we have a function `f(x)` that evaluate the score of a specific result. It's still far for trivial to tell wether one version or another version of the search agent is performing better than the other.
Comments.UserDisplayName: Léonard Henriquez
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-03T08:02:35.813Z
Comments.Text: An output cannot be evaluated in a vaccum. You need to state some kind of measurement, goal or outcome. Often search results are scored arbitrarily on custom expert ranking (e.g. a "relevance" score that might be constructed from number of matches to input text). If yo uare trying to replicate that, but using an AI-powered search, then you can say that. But so far the question literally says "There is no objective metric to determine if one result is better than another." and gives no other clue to what you are doing. This is impossible to answer
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-03T09:23:09.613Z
Comments.Text: The problem was indeed composed of multiple sub problems so I reduced the scope to focus on only one problem. Let's say that there is an evaluation function to score a specific result. If I have this function, how can I assess the performance of my search ?
Comments.UserDisplayName: Léonard Henriquez
Tags: <search><intelligent-agent><evaluation-functions>
Title: How to evaluate the relevance of a search made with an AI Agent?
UserDisplayName: Léonard Henriquez
ThreadText: # How to evaluate the relevance of a search made with an AI Agent?
  
  (Asked by: username_0 on 2024-05-29)
  
  I’m building an AI agent designed to find the best results for with a search tool. The agent is essentially a Large Language Model (LLM) connected to a Search API via function calling. The goal is for this agent to use the Search API to identify the most relevant results.
  
  Let's imagine that I have an evaluation function `f(x)` that allow to score a specific result.
  
  How can I evaluate the performance of my agent ?
  For instance, how can I tell between two versions of my agent (v1 and v2), which one is performing better ?
  
  Here is a specific constraint due to the fact that in a real world search, I don’t have access to the entire database of results due to the sheer volume (millions of results) and the cost involved. Since I can’t purchase all these results, it’s difficult to know if a specific record should be included in the search results. So I can't use recall like metrics.
  
  Given this constraint, how can I effectively evaluate the performance of my AI agent?
  
  **Tags:** search, intelligent-agent, evaluation-functions
  > **Comments:**
  > **username_1**: How do you assess leads found by humans doing the same task?  
  > **username_2**: Looks like an Unsupervised Learning task to me. So you could _cluster_ your leads by some criterion. There are many criteria, and many methods of cluster analysis.  
  > **username_0**: @NeilSlater that’s a very good question. Typically, this process is done through qualitative human evaluation (in other words, it’s not very scientific or rationalized). But that’s precisely my question! How can I turn this manually handled problem into one that can be solved with AI?  
  > **username_1**: That would be useful to know, and helps frame the question as automation of something existing (as opposed to asking for magic, which is close to how the question currently phrases it). Please put some information about how humans performing the same task are assessed into the question. It doesn't have to be detailed, just a variation of your comment. As a spoiler, of course someone answering will be suggesting you assess the AI's outputs in the same way, so you may want to elaborate on the resources you have at your disposal, and what opportunities you have for online feedback.  
  > **username_1**: Another obvious follow-up is do you have data from a history of good and bad leads found by humans to use for training of any assessment function?  
  > **username_0**: @NeilSlater, sorry, my question wasn't clear. I don't need help in finding a business metric to specifically evaluate the quality of a lead. It's more of a generic problem that is "how to evaluate the accuracy of a search done by an AI agent". The lead use case was just to give an example more concrete to the problem. I edited the post to remove mention of the lead use case.  
  > **username_1**: OK, so now we're back to "magic" unless you can explain what you want . . . ignoring the AI for a moment, do you have any way to assess search results from your system? Any AI is just going to be a system that tries to maxmise the results from that metric in a more sophisticated manner than a non-AI-based solution. The metric does not need to be _direct_. It could be based on results such as saving the searchers time, making them happy, or some financially-measured benefit. However the more direct and individually targetted metrics are easier to use for training.  
  > **username_1**: Also the metric does not need to be something assessed on all results forever. You could collect and score data about good and bad search results, and use that to train an agent that you may reasonably *expect* to perform at a certain level. So, do you have any way at all to tell whether one search result is better than another, or to give it a rating, assign a financial benefit etc?  
  > **username_0**: OK @NeilSlater, there nothing related to magic here. I'm asking for a scientific method to evaluate search results. Evaluating search results is a problem like many other in LLM evaluation. Let's say that we have a function `f(x)` that evaluate the score of a specific result. It's still far for trivial to tell wether one version or another version of the search agent is performing better than the other.  
  > **username_1**: An output cannot be evaluated in a vaccum. You need to state some kind of measurement, goal or outcome. Often search results are scored arbitrarily on custom expert ranking (e.g. a "relevance" score that might be constructed from number of matches to input text). If yo uare trying to replicate that, but using an AI-powered search, then you can say that. But so far the question literally says "There is no objective metric to determine if one result is better than another." and gives no other clue to what you are doing. This is impossible to answer  
  > **username_0**: The problem was indeed composed of multiple sub problems so I reduced the scope to focus on only one problem. Let's say that there is an evaluation function to score a specific result. If I have this function, how can I assess the performance of my search ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-03. Score: 0 | Accepted answer
  
  > Let's imagine that I have an evaluation function `f(x)` that allow to score a specific result.
  
  It's not clear from this, whether `f(x)` is a relevence score for a single found item, or is a rating for a complete search result. If it is the former, then you can construct the latter by aggregating results in some semi-arbitrary fashion such as taking an average of `f(x)` for the top ten results returned by a search. This aggregation is arbitrary, but you can base it on intuitive concepts such as the number of results presented in a page. Ideally you have this grounded in some sense relating to the higher level goals of the system (i.e. although the system is a search system, no-one has "searching" as a stand-alone goal in practice, there is always some context).
  
  To further aggregate this value per attempt into a value *per search agent*, you will need to construct a test across multiple requests, covering a gamut of different search terms representative of the kinds of search you expect the system will be used for. Take the mean result or `f(x)` score over all test searches in the test set. This will be necessary because it is very unlikely that different searches will be equally effective, so you will want some kind of averaged score that represents the system's performance against how it will be used in practice.
  
  > For instance, how can I tell between two versions of my agent (v1 and v2), which one is performing better ?
  
  It is the one with the better aggregated `f(x)` score, measured over a standardised test set. You can reduce cost of measuring this, at the expense of accuracy, by using a smaller fair sample from the test set. Or you could achieve similar by having small/fast test set to get rough indication and then a larger test set for refining things. You will want to understand the standard error in your measures of aggregate `f(x)` so you can tell when a difference is large enough to be significant.
  
  > I don’t have access to the entire database of results due to the sheer volume (millions of results) and the cost involved.
  
  Whilst this might rule out certain kinds of system validation, such as demonstrating the current `f(x)` is the best possible, it does not prevent you testing the system statistically. The important detail is to make the results on repeated test runs with different agents *equivalent* in as many ways as possible - changing only the internal and controllable aspects of the agent between tests, not how or what you measure.
  
  
  ---
  
  