title: Atla Selene Mini: A General Purpose Evaluation Model
summary: We introduce Atla Selene Mini, a state-of-the-art small language
  model-as-a-judge (SLMJ). Selene Mini is a general-purpose evaluator that
  outperforms the best SLMJs and GPT-4o-mini on overall performance across 11
  out-of-distribution benchmarks, spanning absolute scoring, classification, and
  pairwise preference tasks. It is the highest-scoring 8B generative model on
  RewardBench, surpassing strong baselines like GPT-4o and specialized judges. To
  achieve this, we develop a principled data curation strategy that augments
  public datasets with synthetically generated critiques and ensures high quality
  through filtering and dataset ablations. We train our model on a combined
  direct preference optimization (DPO) and supervised fine-tuning (SFT) loss, and
  produce a highly promptable evaluator that excels in real-world scenarios.
  Selene Mini shows dramatically improved zero-shot agreement with human expert
  evaluations on financial and medical industry datasets. It is also robust to
  variations in prompt format. Preliminary results indicate that Selene Mini is
  the top-ranking evaluator in a live, community-driven Judge Arena. We release
  the model weights on HuggingFace
  (https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) and Ollama to encourage
  widespread community adoption.
paper_authors._id: 679ae7655c55250b48483742
paper_authors.name: Andrei Alexandru
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T15:17:07.941Z
paper_authors._id: 62571e9e0e0c97db812e3afb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662586587273-62571e9e0e0c97db812e3afb.jpeg
paper_authors.fullname: Andrei Alexandru
paper_authors.type: user
paper_authors.user: inwaves
paper_authors._id: 679ae7655c55250b48483743
paper_authors.name: Antonia Calvi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-30T09:40:54.827Z
paper_authors._id: 66e184e86048d62cd8fb4e52
paper_authors.avatarUrl: /avatars/dc459c692fe9fce0911fa1229df0aeee.svg
paper_authors.fullname: Antonia Calvi
paper_authors.type: user
paper_authors.user: NinaCalvi
paper_authors._id: 679ae7655c55250b48483744
paper_authors.name: Henry Broomfield
paper_authors._id: 679ae7655c55250b48483745
paper_authors.name: Jackson Golden
paper_authors._id: 679ae7655c55250b48483746
paper_authors.name: Kyle Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T15:17:03.347Z
paper_authors._id: 659fc8832cb13cede03047bb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/659fc8832cb13cede03047bb/Wo_LjryGEJFnxXrOcokfE.jpeg
paper_authors.fullname: kyle
paper_authors.type: user
paper_authors.user: kaikaidai
paper_authors._id: 679ae7655c55250b48483747
paper_authors.name: Mathias Leys
paper_authors._id: 679ae7655c55250b48483748
paper_authors.name: Maurice Burger
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T15:17:05.147Z
paper_authors._id: 66d08d5c952f5e4e64bd6be0
paper_authors.avatarUrl: /avatars/2fc3a6e3813718f0c001fb26337dab45.svg
paper_authors.fullname: Maurice
paper_authors.type: user
paper_authors.user: MauriceBurg
paper_authors._id: 679ae7655c55250b48483749
paper_authors.name: Max Bartolo
paper_authors._id: 679ae7655c55250b4848374a
paper_authors.name: Roman Engeler
paper_authors._id: 679ae7655c55250b4848374b
paper_authors.name: Sashank Pisupati
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T15:17:06.579Z
paper_authors._id: 633c4fb100732349209f2aad
paper_authors.avatarUrl: /avatars/b44ccae4fb097284730291e4fcc47a24.svg
paper_authors.fullname: Sashank Pisupati
paper_authors.type: user
paper_authors.user: spisupat
paper_authors._id: 679ae7655c55250b4848374c
paper_authors.name: Toby Drane
paper_authors._id: 679ae7655c55250b4848374d
paper_authors.name: Young Sun Park
paper_title: Atla Selene Mini: A General Purpose Evaluation Model
paper_summary: We introduce Atla Selene Mini, a state-of-the-art small language
  model-as-a-judge (SLMJ). Selene Mini is a general-purpose evaluator that
  outperforms the best SLMJs and GPT-4o-mini on overall performance across 11
  out-of-distribution benchmarks, spanning absolute scoring, classification, and
  pairwise preference tasks. It is the highest-scoring 8B generative model on
  RewardBench, surpassing strong baselines like GPT-4o and specialized judges. To
  achieve this, we develop a principled data curation strategy that augments
  public datasets with synthetically generated critiques and ensures high quality
  through filtering and dataset ablations. We train our model on a combined
  direct preference optimization (DPO) and supervised fine-tuning (SFT) loss, and
  produce a highly promptable evaluator that excels in real-world scenarios.
  Selene Mini shows dramatically improved zero-shot agreement with human expert
  evaluations on financial and medical industry datasets. It is also robust to
  variations in prompt format. Preliminary results indicate that Selene Mini is
  the top-ranking evaluator in a live, community-driven Judge Arena. We release
  the model weights on HuggingFace
  (https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) and Ollama to encourage
  widespread community adoption.
paper_ai_summary: Atla Selene Mini, an 8B language model-as-a-judge, excels across various benchmarks using enhanced data curation and a combined training approach, achieving top performance in zero-shot evaluations and real-world scenarios.
paper_ai_keywords: data curation synthetically generated critiques direct preference optimization supervised fine-tuning promptable evaluator zero-shot agreement human expert evaluations