Answers.Body: <p>I ended up with converting the compares to an absolute scale using something like ELO scoring from chess (<a href="https://en.wikipedia.org/wiki/Elo_rating_system" rel="nofollow noreferrer">Wikipedia</a>, <a href="https://link.springer.com/article/10.3758/s13428-017-0898-2" rel="nofollow noreferrer">a paper</a>), than you can just use correlation as usual.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-03T16:59:23.243Z
Answers.LastActivityDate: 2023-08-03T18:50:48.533Z
Answers.LastEditDate: 2023-08-03T18:50:48.533Z
Answers.UserDisplayName: Osmosis D. Jones
Body: <p>I have built an automatic metric (using DL) to estimate some property of text. I want to evaluate its performance by comparing it to human judgments with respect to that property. As far as I know this is usually done using correlation. I found a related data sets. The problem is that the property is not annotated on an absolute scale but with comparisons.</p>
  <p>Imagine for example something like this:</p>
  <pre><code>Text_A &lt; Text_B &lt; Text_C;
  Text_D &lt; Text_B &lt; Text_D;
  ...
  </code></pre>
  <p>(The text are ranked according to the property. Each Text can occur in multiple compares, but I think not everything is compared with everything.)</p>
  <p>So the question is: <strong>What is the way to get the correlation? (Or what is the best way to judge the alignment?)</strong></p>
  <p>I would be happy about every hint (including suggestions to improve the question)!</p>
  <hr />
  <p>Further Information:</p>
  <ul>
  <li>I assume such data might originate from using Best-worst scaling (but that's not explicitly stated in the source).</li>
  <li>I assume using a rank based correlation fits more than the standard correlation.</li>
  </ul>
  <hr />
  <p>My attempts:</p>
  <ul>
  <li><p>An idea is to <strong>average the correlations</strong> of each set of compares. But I haven't found a way to perform the average in an exact way. Fisher z transformation seems to be only an estimate and as I have a lot of individual correlations with just 3 data points each, I assume the bias would be quite significant. <strong>Does anybody know a way to exactly average correlations for sets of equal size?</strong></p>
  </li>
  <li><p>I was thinking about using <strong>accuracy</strong> instead of correlation, as it seems easier to aggregate. Just convert everything into binary compares:</p>
  <p>Text_A &lt; Text_B &lt; Text_C;</p>
  <p>-&gt;</p>
  <p>Text_A &lt; Text_B; Text_B &lt; Text_C; Text_A &lt; Text_C;</p>
  <p>But I am not sure whether this is appropriated. <strong>What are advantages of accuracy and correlation, why would one prefer one over the other? What is their relationship?</strong></p>
  <p>Using a small R experiment I figured out that accuracy and correlation seem to be closely aligned but not perfect, see <a href="https://github.com/UntotaufUrlaub/cor_vs_acc/blob/main/fiddle.R" rel="nofollow noreferrer">code on github</a>. (The experiments did not consider ties in the the simulated ranking data.)</p>
  </li>
  </ul>
  
Tags: <accuracy><human-like><metric><correlation>
Title: How to determine alignment / correlation with comparison based judgments?
UserDisplayName: Osmosis D. Jones
ThreadText: # How to determine alignment / correlation with comparison based judgments?
  
  (Asked by: username_0 on 2022-12-16)
  
  I have built an automatic metric (using DL) to estimate some property of text. I want to evaluate its performance by comparing it to human judgments with respect to that property. As far as I know this is usually done using correlation. I found a related data sets. The problem is that the property is not annotated on an absolute scale but with comparisons.
  Imagine for example something like this:
  Text_A < Text_B < Text_C;
  Text_D < Text_B < Text_D;
  ...
  
  (The text are ranked according to the property. Each Text can occur in multiple compares, but I think not everything is compared with everything.)
  So the question is: What is the way to get the correlation? (Or what is the best way to judge the alignment?)
  I would be happy about every hint (including suggestions to improve the question)!
  
  Further Information:
  
  I assume such data might originate from using Best-worst scaling (but that's not explicitly stated in the source).
  I assume using a rank based correlation fits more than the standard correlation.
  
  
  My attempts:
  
  An idea is to average the correlations of each set of compares. But I haven't found a way to perform the average in an exact way. Fisher z transformation seems to be only an estimate and as I have a lot of individual correlations with just 3 data points each, I assume the bias would be quite significant. Does anybody know a way to exactly average correlations for sets of equal size?
  
  I was thinking about using accuracy instead of correlation, as it seems easier to aggregate. Just convert everything into binary compares:
  Text_A < Text_B < Text_C;
  ->
  Text_A < Text_B; Text_B < Text_C; Text_A < Text_C;
  But I am not sure whether this is appropriated. What are advantages of accuracy and correlation, why would one prefer one over the other? What is their relationship?
  Using a small R experiment I figured out that accuracy and correlation seem to be closely aligned but not perfect, see code on github. (The experiments did not consider ties in the the simulated ranking data.)
  
  **Tags:** accuracy, human-like, metric, correlation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-08-03. Score: 1
  
  I ended up with converting the compares to an absolute scale using something like ELO scoring from chess ([Wikipedia](https://en.wikipedia.org/wiki/Elo_rating_system), [a paper](https://link.springer.com/article/10.3758/s13428-017-0898-2)), than you can just use correlation as usual.
  
  
  ---
  
  