Answers.Body: <p>Determining the right classification/prediction threshold is always a trade-off between true positives, true negatives, false positives and false negatives. There is no universal guideline for choosing such a threshold as it heavily depends on your data and your model.</p>
  <p>However, there are some tools that might help you get a better idea of how your model operates under different thresholds. For example, you might use an ROC curve (plotting the True positive v False positive rate for various thresholds) or the Precision-Recall curve (plotting Precision v Recall for various thresholds).</p>
  <p>These tools can give you insights into how your model responds to different thresholds and help you choose an appropriate threshold for your application.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T11:23:41.720Z
Answers.LastActivityDate: 2023-05-26T11:23:41.720Z
Answers.UserDisplayName: Robin van Hoorn
Answers.Body: <p>As correctly explained by @Robin van Hoorn, determining the classification threshold involves a trade-off between correct predictions and errors.</p>
  <p>One approach is to consider the <strong>TPR</strong> (true positive rate), saying that for your application you want to achieve <span class="math-container">$N\%$</span> of TPR (i.e. true class correctly predicted <span class="math-container">$N/100$</span> times), e.g. <span class="math-container">$N=90\%$</span>. You can compute the TPR (e.g. with <code>scikit-learn</code>) and then look for the threshold value that achieves the desired rate of TPR: that would be your classification threshold.</p>
  <p>Similarly, you can reason on the errors the model do and look instead at the <strong>FPR</strong>, where FPR is the false positive rate. This time the interpretation is to let the model do at most <span class="math-container">$N\%$</span> misclassifications, you you want to limit errors.</p>
  <p>Indeed, the right value of <span class="math-container">$N$</span> is application and problem specific but can be determined by visual inspection of the ROC curve. The ROC compares TPR against FPR, and you can image in which place of the curve you want your classifier to be. Now, look at the axis and you'll get the respective value of TPR. Lastly, use that value to compute the threshold (which should be done on a validation set.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T17:49:16.783Z
Answers.LastActivityDate: 2023-05-26T17:49:16.783Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am using openai's <code>text-embedding-ada-002</code> embeddings model to do a semantic search on a database of articles to find articles that are most related to a given input text. I am looking for a way to define a minimum similarity score to prevent returning articles that aren't actually related enough.</p>
  <p>There is two difficulties that I have:</p>
  <ol>
  <li>For some search queries a certain similarity score seems appropriate as a minimum treshold value, but then for others that minimum value seems to be too strict. For instance I find that for very well defined specific topics you generally want a higher treshold similarity score than for more broad or generic texts. That's my intuition so far at least.</li>
  <li>The scores of the openai embedding model almost always fall between 0.77 and 1 instead of using the entire range of -1 to 1 and in reality the scores in normal cases all fall around 0.88. Having all scores so close to eachother makes it harder to pinpoint a good treshold value.</li>
  </ol>
  <p>Are any known methods for determining a good treshold value for cosine similarity scores?</p>
  
Tags: <natural-language-processing><classification><embeddings><cosine-similarity>
Title: How do I choose a good treshold for classification (using cosine similarity scores)?
UserDisplayName: Stefan
ThreadText: # How do I choose a good treshold for classification (using cosine similarity scores)?
  
  (Asked by: username_0 on 2023-05-26)
  
  I am using openai's `text-embedding-ada-002` embeddings model to do a semantic search on a database of articles to find articles that are most related to a given input text. I am looking for a way to define a minimum similarity score to prevent returning articles that aren't actually related enough.
  
  There is two difficulties that I have:
  
  1. For some search queries a certain similarity score seems appropriate as a minimum treshold value, but then for others that minimum value seems to be too strict. For instance I find that for very well defined specific topics you generally want a higher treshold similarity score than for more broad or generic texts. That's my intuition so far at least.
  2. The scores of the openai embedding model almost always fall between 0.77 and 1 instead of using the entire range of -1 to 1 and in reality the scores in normal cases all fall around 0.88. Having all scores so close to eachother makes it harder to pinpoint a good treshold value.
  
  Are any known methods for determining a good treshold value for cosine similarity scores?
  
  **Tags:** natural-language-processing, classification, embeddings, cosine-similarity
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-05-26. Score: 2
  
  Determining the right classification/prediction threshold is always a trade-off between true positives, true negatives, false positives and false negatives. There is no universal guideline for choosing such a threshold as it heavily depends on your data and your model.
  
  However, there are some tools that might help you get a better idea of how your model operates under different thresholds. For example, you might use an ROC curve (plotting the True positive v False positive rate for various thresholds) or the Precision-Recall curve (plotting Precision v Recall for various thresholds).
  
  These tools can give you insights into how your model responds to different thresholds and help you choose an appropriate threshold for your application.
  
  
  ---
  
  ### Answer by username_2 on 2023-05-26. Score: 2
  
  As correctly explained by @username_1, determining the classification threshold involves a trade-off between correct predictions and errors.
  
  One approach is to consider the **TPR** (true positive rate), saying that for your application you want to achieve $N\%$ of TPR (i.e. true class correctly predicted $N/100$ times), e.g. $N=90\%$. You can compute the TPR (e.g. with `scikit-learn`) and then look for the threshold value that achieves the desired rate of TPR: that would be your classification threshold.
  
  Similarly, you can reason on the errors the model do and look instead at the **FPR**, where FPR is the false positive rate. This time the interpretation is to let the model do at most $N\%$ misclassifications, you you want to limit errors.
  
  Indeed, the right value of $N$ is application and problem specific but can be determined by visual inspection of the ROC curve. The ROC compares TPR against FPR, and you can image in which place of the curve you want your classifier to be. Now, look at the axis and you'll get the respective value of TPR. Lastly, use that value to compute the threshold (which should be done on a validation set.)
  
  
  ---
  
  