title: Reliable, Reproducible, and Really Fast Leaderboards with Evalica
summary: The rapid advancement of natural language processing (NLP) technologies, such
  as instruction-tuned large language models (LLMs), urges the development of
  modern evaluation protocols with human and machine feedback. We introduce
  Evalica, an open-source toolkit that facilitates the creation of reliable and
  reproducible model leaderboards. This paper presents its design, evaluates its
  performance, and demonstrates its usability through its Web interface,
  command-line interface, and Python API.
paper_authors._id: 67613d66896c29e2a598e378
paper_authors.name: Dmitry Ustalov
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-17T08:59:57.545Z
paper_authors._id: 60ef34ff8432bc401cd0abbf
paper_authors.avatarUrl: /avatars/4992b27a8230b80e9bb9a4f92bd5638c.svg
paper_authors.fullname: Dmitry Ustalov
paper_authors.type: user
paper_authors.user: dustalov
paper_title: Reliable, Reproducible, and Really Fast Leaderboards with Evalica
paper_summary: The rapid advancement of natural language processing (NLP) technologies, such
  as instruction-tuned large language models (LLMs), urges the development of
  modern evaluation protocols with human and machine feedback. We introduce
  Evalica, an open-source toolkit that facilitates the creation of reliable and
  reproducible model leaderboards. This paper presents its design, evaluates its
  performance, and demonstrates its usability through its Web interface,
  command-line interface, and Python API.
paper_ai_summary: Evalica is an open-source toolkit designed to create reliable and reproducible leaderboards for instruction-tuned large language models with support for human and machine feedback.
paper_ai_keywords: instruction-tuned large language models Evalica model leaderboards Web interface command-line interface Python API