models can be seen in Table 1. We note that human and GPT-4 ranking of models on the Vicuna
benchmark disagree partially, particularly for Guanaco 7B, but are consistent for most models with
a Kendall Tau of τ = 0.43 and Spearman rank correlation of r = 0.55 at the system level. At the
example level, the agreement between GPT-4 and human annotators’ majority vote is weaker with
Fleiss κ = 0.25. Overall, this shows a moderate agreement between system-level judgments by
GPT-4 and human annotators, and thus that model-based evaluation represents a somewhat reliable
alternative to human evaluation. We discuss further considerations in Section 6.2.
Elo rankings in Table 7 indicate that Guanaco 33B and 65B models outperform all models besides
GPT-4 on the Vicuna and OA benchmarks and that they perform comparably to ChatGPT in line
with Table 6. We note that the Vicuna benchmark favors open-source models while the larger OA
QLORA