control for ordering effects, e.g., the effect of which model pairs compete with each other first.
5.3 Guanaco: QLORA trained on OASST1 is a State-of-the-art Chatbot
Based on our automated and human evaluations, we find that the top QLORA tuned model, Guanaco
65B, which we finetune on a variant of OASST1, is the best-performing open-source chatbot model
and offers performance competitive to ChatGPT. When compared to GPT-4, Guanaco 65B and 33B
have an expected win probability of 30%, based on Elo rating from human annotators system-level
pairwise comparisons - the highest reported to date.
The Vicuna benchmark [10] results relative to ChatGPT are shown in Table 6. We find that Guanaco
65B is the best-performing model after GPT-4, achieving 99.3% performance relative to ChatGPT.
Guanaco 33B has more parameters than the Vicuna 13B model, but uses only 4-bit precision for its
weights and is thus much more memory efficient at 21 GB vs 26 GB, providing a three percentage
QLORA