a model is calculated as a percentage of the score that ChatGPT achieved. Note this relative score
can be higher than 100% if the model achieves a higher absolute score than ChatGPT. We find a
significant ordering effect with GPT-4 increasing the score of the response occurring earlier in the
prompt. To control for such effects, we recommend reporting the mean score over both orders.
Next, we measure performance through direct comparisons between system outputs. We simplify
the rating scheme to a three-class labeling problem that accounts for ties. We prompt GPT-4 to
pick the best response or declare a tie and provide an explanation. We conduct these head-to-head
comparisons on all permutations of pairs of systems on both the Vicuna and OA benchmarks.
Human Evaluation While recent work indicates generative models can be effectively employed
for system evaluations [19], the reliability GPT-4 ratings to assess chatbot performance is, to our
QLORA