learning rate for 33B and 65B while doubling the batch size.
Baselines We compare our models to both research (Vicuna [10] and Open Assistant [31]) and
commercial (GPT-4 [42], GPT-3.5-turbo and Bard) chatbot systems. The Open Assistant model is
a LLaMA 33B model finetuned with Reinforcement Learning from Human Feedback (RLHF) on
the same OASST1 dataset that we experiment with. Vicuna does full fine-tuning of LLaMA 13B
on proprietary user-shared conversations from ShareGPT and is thus the result of distillation from
OpenAI GPT models.
5.2 Evaluation
Following common practice, we use the MMLU (Mas-
sively Multitask Language Understanding) benchmark
[24] to measure performance on a range of language un-
derstanding tasks. This is a multiple-choice benchmark
covering 57 tasks including elementary mathematics,
US history, computer science, law, and more. We report
5-shot test accuracy.
We also test generative language capabilities through
both automated and human evaluations. This second
QLORA