Furthermore, we also provide a extensive analysis of chatbot performance that uses both human
raters and GPT-4 for evaluation. We use tournament-style benchmarking where models compete
against each other in matches to produce the best response for a given prompt. The winner of a
match is judged by either GPT-4 or human annotators. The tournament results are aggregated into
Elo scores [16, 17] which determine the ranking of chatbot performance. We find that GPT-4 and
human evaluations largely agree on the rank of model performance in the tournaments, but we also
find there are instances of strong disagreement. As such, we highlight that model-based evaluation
while providing a cheap alternative to human-annotation also has its uncertainties.
We augment our chatbot benchmark results with a qualitative analysis of Guanaco models. Our analy-
sis highlights success and failure cases that were not captured by the quantitative benchmarks.
QLORA