title: SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal
    Domain
summary: In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language
  models (LLMs) tailored for the legal sector. These models, which feature
  architectures of 54 billion and 141 billion parameters, respectively, are based
  on the Mixtral architecture. The development of SaulLM-54B and SaulLM-141B is
  guided by large-scale domain adaptation, divided into three strategies: (1) the
  exploitation of continued pretraining involving a base corpus that includes
  over 540 billion of legal tokens, (2) the implementation of a specialized legal
  instruction-following protocol, and (3) the alignment of model outputs with
  human preferences in legal interpretations. The integration of synthetically
  generated data in the second and third steps enhances the models' capabilities
  in interpreting and processing legal texts, effectively reaching
  state-of-the-art performance and outperforming previous open-source models on
  LegalBench-Instruct. This work explores the trade-offs involved in
  domain-specific adaptation at this scale, offering insights that may inform
  future studies on domain adaptation using strong decoder models. Building upon
  SaulLM-7B, this study refines the approach to produce an LLM better equipped
  for legal tasks. We are releasing base, instruct, and aligned versions on top
  of SaulLM-54B and SaulLM-141B under the MIT License to facilitate reuse and
  collaborative research.
paper_authors._id: 66a8439359d86126000b54cc
paper_authors.name: Pierre Colombo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:36:24.366Z
paper_authors._id: 644a900e3a619fe72b14af0f
paper_authors.avatarUrl: /avatars/e2d5dac3d92757ed48e37e126a3464a3.svg
paper_authors.fullname: Colombo
paper_authors.type: user
paper_authors.user: PierreColombo
paper_authors._id: 66a8439359d86126000b54cd
paper_authors.name: Telmo Pires
paper_authors._id: 66a8439359d86126000b54ce
paper_authors.name: Malik Boudiaf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:36:48.139Z
paper_authors._id: 639283dd2c0c90e2f31b5500
paper_authors.avatarUrl: /avatars/9a3cff6b14fd687bcbe2b78c4c032619.svg
paper_authors.fullname: Malik Boudiaf
paper_authors.type: user
paper_authors.user: mboudiaf
paper_authors._id: 66a8439359d86126000b54cf
paper_authors.name: Rui Melo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:36:53.989Z
paper_authors._id: 622f26eb163b4ef6fd7a5361
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/622f26eb163b4ef6fd7a5361/-WYJwPC5QkWecz_XL8w94.jpeg
paper_authors.fullname: Rui Melo
paper_authors.type: user
paper_authors.user: rufimelo
paper_authors._id: 66a8439359d86126000b54d0
paper_authors.name: Dominic Culver
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:36:59.970Z
paper_authors._id: 6453da2ef2122886b298873f
paper_authors.avatarUrl: /avatars/8c79e43e5d76cd8d29db9357180b629d.svg
paper_authors.fullname: Dominic Culver
paper_authors.type: user
paper_authors.user: CWcx
paper_authors._id: 66a8439359d86126000b54d1
paper_authors.name: Sofia Morgado
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:37:05.378Z
paper_authors._id: 63e76a3ce02ee67e8e4fcdc2
paper_authors.avatarUrl: /avatars/ca8668fb2fdafb264291a7585c0ddd6b.svg
paper_authors.fullname: Sofia Morgado
paper_authors.type: user
paper_authors.user: somorgado
paper_authors._id: 66a8439359d86126000b54d2
paper_authors.name: Etienne Malaboeuf
paper_authors._id: 66a8439359d86126000b54d3
paper_authors.name: Gabriel Hautreux
paper_authors._id: 66a8439359d86126000b54d4
paper_authors.name: Johanne Charpentier
paper_authors._id: 66a8439359d86126000b54d5
paper_authors.name: Michael Desa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:38:05.700Z
paper_authors._id: 62cee925e7f6014c0e9d7b93
paper_authors.avatarUrl: /avatars/39e82af1550d57d06963ad87d42514e5.svg
paper_authors.fullname: Michael Desa
paper_authors.type: user
paper_authors.user: desa
paper_title: SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal
    Domain
paper_summary: In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language
  models (LLMs) tailored for the legal sector. These models, which feature
  architectures of 54 billion and 141 billion parameters, respectively, are based
  on the Mixtral architecture. The development of SaulLM-54B and SaulLM-141B is
  guided by large-scale domain adaptation, divided into three strategies: (1) the
  exploitation of continued pretraining involving a base corpus that includes
  over 540 billion of legal tokens, (2) the implementation of a specialized legal
  instruction-following protocol, and (3) the alignment of model outputs with
  human preferences in legal interpretations. The integration of synthetically
  generated data in the second and third steps enhances the models' capabilities
  in interpreting and processing legal texts, effectively reaching
  state-of-the-art performance and outperforming previous open-source models on
  LegalBench-Instruct. This work explores the trade-offs involved in
  domain-specific adaptation at this scale, offering insights that may inform
  future studies on domain adaptation using strong decoder models. Building upon
  SaulLM-7B, this study refines the approach to produce an LLM better equipped
  for legal tasks. We are releasing base, instruct, and aligned versions on top
  of SaulLM-54B and SaulLM-141B under the MIT License to facilitate reuse and
  collaborative research.
paper_ai_summary: Two large legal language models, SaulLM-54B and SaulLM-141B, based on the Mixtral architecture, are introduced for domain-specific adaptation in the legal sector using continued pretraining, specialized protocols, and preference alignment with synthetic data.
paper_ai_keywords: Mixtral architecture large language models legal sector domain adaptation continued pretraining legal tokens legal instruction-following protocol human preferences legal interpretations synthetic data LegalBench-Instruct strong decoder models