title: SwiLTra-Bench: The Swiss Legal Translation Benchmark
summary: In Switzerland legal translation is uniquely important due to the country's
  four official languages and requirements for multilingual legal documentation.
  However, this process traditionally relies on professionals who must be both
  legal experts and skilled translators -- creating bottlenecks and impacting
  effective access to justice. To address this challenge, we introduce
  SwiLTra-Bench, a comprehensive multilingual benchmark of over 180K aligned
  Swiss legal translation pairs comprising laws, headnotes, and press releases
  across all Swiss languages along with English, designed to evaluate LLM-based
  translation systems. Our systematic evaluation reveals that frontier models
  achieve superior translation performance across all document types, while
  specialized translation systems excel specifically in laws but under-perform in
  headnotes. Through rigorous testing and human expert validation, we demonstrate
  that while fine-tuning open SLMs significantly improves their translation
  quality, they still lag behind the best zero-shot prompted frontier models such
  as Claude-3.5-Sonnet. Additionally, we present SwiLTra-Judge, a specialized LLM
  evaluation system that aligns best with human expert assessments.
paper_authors._id: 67c6bd6e8f3e7fd471affd06
paper_authors.name: Joel Niklaus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:39:30.778Z
paper_authors._id: 5fae5f68b8423e1d80b8a988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656078368144-5fae5f68b8423e1d80b8a988.jpeg
paper_authors.fullname: Joel Niklaus
paper_authors.type: user
paper_authors.user: joelniklaus
paper_authors._id: 67c6bd6e8f3e7fd471affd07
paper_authors.name: Jakob Merane
paper_authors._id: 67c6bd6e8f3e7fd471affd08
paper_authors.name: Luka Nenadic
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:39:40.122Z
paper_authors._id: 66b46ea33a88f464e2842ee8
paper_authors.avatarUrl: /avatars/a204b5828a96cf0d5334f238c6c96b13.svg
paper_authors.fullname: Luka Nenadic
paper_authors.type: user
paper_authors.user: lukanen
paper_authors._id: 67c6bd6e8f3e7fd471affd09
paper_authors.name: Sina Ahmadi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:39:55.817Z
paper_authors._id: 647530b3e0b188d3cb24b81b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647530b3e0b188d3cb24b81b/JzAMifLHLXhjkRSKeF5iQ.jpeg
paper_authors.fullname: Sina Ahm
paper_authors.type: user
paper_authors.user: SinaAhmadi
paper_authors._id: 67c6bd6e8f3e7fd471affd0a
paper_authors.name: Yingqiang Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:40:05.409Z
paper_authors._id: 645e1d5dad4ded42382ab518
paper_authors.avatarUrl: /avatars/a33339d37e0ada45198452e48a918be2.svg
paper_authors.fullname: Yingqiang Gao
paper_authors.type: user
paper_authors.user: CharizardAcademy
paper_authors._id: 67c6bd6e8f3e7fd471affd0b
paper_authors.name: Cyrill A. H. Chevalley
paper_authors._id: 67c6bd6e8f3e7fd471affd0c
paper_authors.name: Claude Humbel
paper_authors._id: 67c6bd6e8f3e7fd471affd0d
paper_authors.name: Christophe Gösken
paper_authors._id: 67c6bd6e8f3e7fd471affd0e
paper_authors.name: Lorenzo Tanzi
paper_authors._id: 67c6bd6e8f3e7fd471affd0f
paper_authors.name: Thomas Lüthi
paper_authors._id: 67c6bd6e8f3e7fd471affd10
paper_authors.name: Stefan Palombo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:52:02.245Z
paper_authors._id: 6594ee0f57a556fbe1720c37
paper_authors.avatarUrl: /avatars/662e965fbde86e53b86fd9c2c9dc325a.svg
paper_authors.fullname: Stefan Palombo
paper_authors.type: user
paper_authors.user: spalombo
paper_authors._id: 67c6bd6e8f3e7fd471affd11
paper_authors.name: Spencer Poff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:57:18.336Z
paper_authors._id: 63c5cf2176c803a693f59763
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678689670011-63c5cf2176c803a693f59763.jpeg
paper_authors.fullname: Spencer Poff
paper_authors.type: user
paper_authors.user: spencerp
paper_authors._id: 67c6bd6e8f3e7fd471affd12
paper_authors.name: Boling Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:57:11.493Z
paper_authors._id: 65d505dd1e2597ff935d3b03
paper_authors.avatarUrl: /avatars/8f7f2bdf0e93a2442b993d064983126f.svg
paper_authors.fullname: Boling Yang
paper_authors.type: user
paper_authors.user: BolingYang
paper_authors._id: 67c6bd6e8f3e7fd471affd13
paper_authors.name: Nan Wu
paper_authors._id: 67c6bd6e8f3e7fd471affd14
paper_authors.name: Matthew Guillod
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:57:04.689Z
paper_authors._id: 65d3ae44ea28ba508b5c62fd
paper_authors.avatarUrl: /avatars/590cbb68787bd2706a2c2dd711274d61.svg
paper_authors.fullname: Matthew Guillod
paper_authors.type: user
paper_authors.user: mguillod
paper_authors._id: 67c6bd6e8f3e7fd471affd15
paper_authors.name: Robin Mamié
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:56:57.175Z
paper_authors._id: 64df37df92f3bd97867e2122
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64df37df92f3bd97867e2122/IfXh4jk5QZ_Pc6tlH8JqY.jpeg
paper_authors.fullname: Robin Mamié
paper_authors.type: user
paper_authors.user: robinmamie
paper_authors._id: 67c6bd6e8f3e7fd471affd16
paper_authors.name: Daniel Brunner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:56:50.083Z
paper_authors._id: 6630002ad2f21fa96c0780f2
paper_authors.avatarUrl: /avatars/65b33ed0f4afb038fe23ebd98c655a53.svg
paper_authors.fullname: Daniel Brunner
paper_authors.type: user
paper_authors.user: ravecs
paper_authors._id: 67c6bd6e8f3e7fd471affd17
paper_authors.name: Julio Pereyra
paper_authors._id: 67c6bd6e8f3e7fd471affd18
paper_authors.name: Niko Grupen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T13:56:18.392Z
paper_authors._id: 642c89fb3a6a3fa1ebaf95bc
paper_authors.avatarUrl: /avatars/9a1da6fb5e0bf732e1ac29d0722f46db.svg
paper_authors.fullname: Niko Grupen
paper_authors.type: user
paper_authors.user: nikogrupen
paper_title: SwiLTra-Bench: The Swiss Legal Translation Benchmark
paper_summary: In Switzerland legal translation is uniquely important due to the country's
  four official languages and requirements for multilingual legal documentation.
  However, this process traditionally relies on professionals who must be both
  legal experts and skilled translators -- creating bottlenecks and impacting
  effective access to justice. To address this challenge, we introduce
  SwiLTra-Bench, a comprehensive multilingual benchmark of over 180K aligned
  Swiss legal translation pairs comprising laws, headnotes, and press releases
  across all Swiss languages along with English, designed to evaluate LLM-based
  translation systems. Our systematic evaluation reveals that frontier models
  achieve superior translation performance across all document types, while
  specialized translation systems excel specifically in laws but under-perform in
  headnotes. Through rigorous testing and human expert validation, we demonstrate
  that while fine-tuning open SLMs significantly improves their translation
  quality, they still lag behind the best zero-shot prompted frontier models such
  as Claude-3.5-Sonnet. Additionally, we present SwiLTra-Judge, a specialized LLM
  evaluation system that aligns best with human expert assessments.
paper_ai_summary: A multilingual benchmark for Swiss legal translation evaluates various models, revealing that while fine-tuned systems improve performance, they do not match zero-shot models like Claude-3.5-Sonnet, particularly in specialized document types.
paper_ai_keywords: multilingual benchmark Swiss legal translation pairs LLM-based translation systems frontier models specialized translation systems fine-tuning open SLMs zero-shot prompted frontier models SwiLTra-Judge