LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large
  Language Models
Automatic side-by-side evaluation has emerged as a promising approach to
evaluating the quality of responses from large language models (LLMs). However,
analyzing the results from this evaluation approach raises scalability and
interpretability challenges. In this paper, we present LLM Comparator, a novel
visual analytics tool for interactively analyzing results from automatic
side-by-side evaluation. The tool supports interactive workflows for users to
understand when and why a model performs better or worse than a baseline model,
and how the responses from two models are qualitatively different. We
iteratively designed and developed the tool by closely working with researchers
and engineers at a large technology company. This paper details the user
challenges we identified, the design and development of the tool, and an
observational study with participants who regularly evaluate their models.
65d2dac46f1a8bf6a1a1f1bd
Minsuk Kahng
extracted_confirmed
2024-02-19T06:42:36.747Z
6301f3fb4ccccaa23d38fb7f
/avatars/4226111991d1788d3d442ffb2f33ea48.svg
Minsuk Kahng
user
minsukkahng
65d2dac46f1a8bf6a1a1f1be
Ian Tenney
admin_assigned
2024-02-19T11:21:49.868Z
615b839cae7584e90d765049
https://cdn-avatars.huggingface.co/v1/production/uploads/1633387368708-noauth.jpeg
Ian Tenney
user
iftenney
65d2dac46f1a8bf6a1a1f1bf
Mahima Pushkarna
65d2dac46f1a8bf6a1a1f1c0
Michael Xieyang Liu
admin_assigned
2024-02-19T11:22:21.511Z
648e688059c4e5c87dbb4c73
/avatars/c4100b13b8f915dba694908d85cd4c67.svg
Michael Xieyang Liu
user
lxieyang
65d2dac46f1a8bf6a1a1f1c1
James Wexler
extracted_confirmed
2024-02-20T13:50:08.219Z
615afcd8a9c650b10473d55c
/avatars/4904494c8d1a91ed5b45c968580a7e05.svg
James Wexler
user
jwexler
65d2dac46f1a8bf6a1a1f1c2
Emily Reif
extracted_confirmed
2024-02-20T18:12:25.131Z
646b88fb942fd7c135dc0ec1
/avatars/0ebcb6cc72458b947f034ac8e13611e8.svg
Emily Reif
user
emilyreif
65d2dac46f1a8bf6a1a1f1c3
Krystal Kallarackal
65d2dac46f1a8bf6a1a1f1c4
Minsuk Chang
admin_assigned
2024-02-19T11:22:43.685Z
60d34c780c54a218bc775298
/avatars/053fe38dd66c1812a3d9d34b41454c9b.svg
Minsuk Chang
user
minsukchang
65d2dac46f1a8bf6a1a1f1c5
Michael Terry
admin_assigned
2024-02-19T11:22:05.745Z
615b839cae7584e90d765049
https://cdn-avatars.huggingface.co/v1/production/uploads/1633387368708-noauth.jpeg
Ian Tenney
user
iftenney
65d2dac46f1a8bf6a1a1f1c6
Lucas Dixon
admin_assigned
2024-02-19T11:24:20.788Z
64c25a9c2b1ffc2eecbce31d
https://cdn-avatars.huggingface.co/v1/production/uploads/64c25a9c2b1ffc2eecbce31d/uTp3yOPdmCSAmhzWwKUjK.png
Lucas Dixon
user
iislucas
LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large
  Language Models
Automatic side-by-side evaluation has emerged as a promising approach to
evaluating the quality of responses from large language models (LLMs). However,
analyzing the results from this evaluation approach raises scalability and
interpretability challenges. In this paper, we present LLM Comparator, a novel
visual analytics tool for interactively analyzing results from automatic
side-by-side evaluation. The tool supports interactive workflows for users to
understand when and why a model performs better or worse than a baseline model,
and how the responses from two models are qualitatively different. We
iteratively designed and developed the tool by closely working with researchers
and engineers at a large technology company. This paper details the user
challenges we identified, the design and development of the tool, and an
observational study with participants who regularly evaluate their models.
LLM Comparator is a visual analytics tool for interactive evaluation of large language models' performance compared to baselines, addressing scalability and interpretability challenges.
large language models (LLMs) automatic side-by-side evaluation visual analytics interactive workflows qualitative differences observational study