Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge
Aparna Elangovan Lei Xu Jongwoo Ko Mahsa Elyasi Ling Liu Sravan Babu Bodapati Dan Roth
The effectiveness of automatic evaluation of generative models is typically measured by comparing the labels  generated via automation  with  human labels  using correlation metrics. 
However, metrics like Krippendorff's $\alpha$ and Randolph's $\kappa$ were originally designed to measure the reliability of human labeling, thus make assumptions about typical human labeling behavior, and these assumptions may not be applicable to machine generated labels. 
In this paper, we show how *relying on a single aggregate correlation score* can obscure fundamental differences between human  labels and those from automatic evaluation, including LLM-as-a-Judge. 
Specifically, we demonstrate that when the proportion of samples with variation or uncertainty in human assigned labels  is relatively high, machine labels (generated by automatic evaluation methods) may superficially appear to have similar or better correlation with the human majority label compared to the human-to-human (HH) correlation. 
This can create the illusion that labels from automatic evaluation  approximates the human majority label. 
However, as the proportion of samples with consistent human labels increases, the correlation between machine  and human  labels fall well below HH correlation. 
Based on these findings, we first propose  *stratifying data by human label uncertainty* to provide a more robust analysis of automatic evaluation performance. Second, recognizing that uncertainty and variation are inherent in perception-based human evaluations, such as those involving attitudes or preferences, we introduce a new metric -*binned Jensen-Shannon Divergence for perception* for such scenarios to better measure the effectiveness of  automatic evaluations. Third, we present visualization techniques -- *perception charts*, to contextualize correlation measures appropriately and to show the strengths and limitations of automatic evaluation.  We have open-sourced our analysis and visualization tools at https://github.com/amazon-science/BeyondCorrelation.
Automated evaluation, LLM as a judge, correlation measures
Impact of noise when measuring effectiveness of automated evaluation