title: LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large
    Language Models
summary: Automatic side-by-side evaluation has emerged as a promising approach to
  evaluating the quality of responses from large language models (LLMs). However,
  analyzing the results from this evaluation approach raises scalability and
  interpretability challenges. In this paper, we present LLM Comparator, a novel
  visual analytics tool for interactively analyzing results from automatic
  side-by-side evaluation. The tool supports interactive workflows for users to
  understand when and why a model performs better or worse than a baseline model,
  and how the responses from two models are qualitatively different. We
  iteratively designed and developed the tool by closely working with researchers
  and engineers at a large technology company. This paper details the user
  challenges we identified, the design and development of the tool, and an
  observational study with participants who regularly evaluate their models.
paper_authors._id: 65d2dac46f1a8bf6a1a1f1bd
paper_authors.name: Minsuk Kahng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-19T06:42:36.747Z
paper_authors._id: 6301f3fb4ccccaa23d38fb7f
paper_authors.avatarUrl: /avatars/4226111991d1788d3d442ffb2f33ea48.svg
paper_authors.fullname: Minsuk Kahng
paper_authors.type: user
paper_authors.user: minsukkahng
paper_authors._id: 65d2dac46f1a8bf6a1a1f1be
paper_authors.name: Ian Tenney
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:21:49.868Z
paper_authors._id: 615b839cae7584e90d765049
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633387368708-noauth.jpeg
paper_authors.fullname: Ian Tenney
paper_authors.type: user
paper_authors.user: iftenney
paper_authors._id: 65d2dac46f1a8bf6a1a1f1bf
paper_authors.name: Mahima Pushkarna
paper_authors._id: 65d2dac46f1a8bf6a1a1f1c0
paper_authors.name: Michael Xieyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:22:21.511Z
paper_authors._id: 648e688059c4e5c87dbb4c73
paper_authors.avatarUrl: /avatars/c4100b13b8f915dba694908d85cd4c67.svg
paper_authors.fullname: Michael Xieyang Liu
paper_authors.type: user
paper_authors.user: lxieyang
paper_authors._id: 65d2dac46f1a8bf6a1a1f1c1
paper_authors.name: James Wexler
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-20T13:50:08.219Z
paper_authors._id: 615afcd8a9c650b10473d55c
paper_authors.avatarUrl: /avatars/4904494c8d1a91ed5b45c968580a7e05.svg
paper_authors.fullname: James Wexler
paper_authors.type: user
paper_authors.user: jwexler
paper_authors._id: 65d2dac46f1a8bf6a1a1f1c2
paper_authors.name: Emily Reif
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-20T18:12:25.131Z
paper_authors._id: 646b88fb942fd7c135dc0ec1
paper_authors.avatarUrl: /avatars/0ebcb6cc72458b947f034ac8e13611e8.svg
paper_authors.fullname: Emily Reif
paper_authors.type: user
paper_authors.user: emilyreif
paper_authors._id: 65d2dac46f1a8bf6a1a1f1c3
paper_authors.name: Krystal Kallarackal
paper_authors._id: 65d2dac46f1a8bf6a1a1f1c4
paper_authors.name: Minsuk Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:22:43.685Z
paper_authors._id: 60d34c780c54a218bc775298
paper_authors.avatarUrl: /avatars/053fe38dd66c1812a3d9d34b41454c9b.svg
paper_authors.fullname: Minsuk Chang
paper_authors.type: user
paper_authors.user: minsukchang
paper_authors._id: 65d2dac46f1a8bf6a1a1f1c5
paper_authors.name: Michael Terry
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:22:05.745Z
paper_authors._id: 615b839cae7584e90d765049
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633387368708-noauth.jpeg
paper_authors.fullname: Ian Tenney
paper_authors.type: user
paper_authors.user: iftenney
paper_authors._id: 65d2dac46f1a8bf6a1a1f1c6
paper_authors.name: Lucas Dixon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:24:20.788Z
paper_authors._id: 64c25a9c2b1ffc2eecbce31d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c25a9c2b1ffc2eecbce31d/uTp3yOPdmCSAmhzWwKUjK.png
paper_authors.fullname: Lucas Dixon
paper_authors.type: user
paper_authors.user: iislucas
paper_title: LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large
    Language Models
paper_summary: Automatic side-by-side evaluation has emerged as a promising approach to
  evaluating the quality of responses from large language models (LLMs). However,
  analyzing the results from this evaluation approach raises scalability and
  interpretability challenges. In this paper, we present LLM Comparator, a novel
  visual analytics tool for interactively analyzing results from automatic
  side-by-side evaluation. The tool supports interactive workflows for users to
  understand when and why a model performs better or worse than a baseline model,
  and how the responses from two models are qualitatively different. We
  iteratively designed and developed the tool by closely working with researchers
  and engineers at a large technology company. This paper details the user
  challenges we identified, the design and development of the tool, and an
  observational study with participants who regularly evaluate their models.
paper_ai_keywords: LLM Comparator visual analytics tool interactive workflows baseline model qualitative differences