title: Llemma: An Open Language Model For Mathematics
summary: We present Llemma, a large language model for mathematics. We continue
  pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web
  data containing mathematics, and mathematical code, yielding Llemma. On the
  MATH benchmark Llemma outperforms all known open base models, as well as the
  unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is
  capable of tool use and formal theorem proving without any further finetuning.
  We openly release all artifacts, including 7 billion and 34 billion parameter
  models, the Proof-Pile-2, and code to replicate our experiments.
paper_authors._id: 652de6c2d2937a53ed729bfb
paper_authors.name: Zhangir Azerbayev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:47:57.624Z
paper_authors._id: 62cf683454477562dcbd8bf9
paper_authors.avatarUrl: /avatars/404a8913f1bc5faa0e2b2ce767e0078e.svg
paper_authors.fullname: Zhangir Azerbayev
paper_authors.type: user
paper_authors.user: zhangirazerbayev
paper_authors._id: 652de6c2d2937a53ed729bfc
paper_authors.name: Hailey Schoelkopf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:48:06.895Z
paper_authors._id: 62895a0215aeee85756062c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669665010552-62895a0215aeee85756062c4.jpeg
paper_authors.fullname: Hailey Schoelkopf
paper_authors.type: user
paper_authors.user: hails
paper_authors._id: 652de6c2d2937a53ed729bfd
paper_authors.name: Keiran Paster
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:48:14.334Z
paper_authors._id: 62d1cea65f9242437590d7e9
paper_authors.avatarUrl: /avatars/8983669dbe58b1d9310843b34265b673.svg
paper_authors.fullname: Keiran Paster
paper_authors.type: user
paper_authors.user: keirp
paper_authors._id: 652de6c2d2937a53ed729bfe
paper_authors.name: Marco Dos Santos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:48:21.649Z
paper_authors._id: 64146fc68ebec947b8c2fddc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64146fc68ebec947b8c2fddc/tTTdYhcg5cp2U8LGZustC.png
paper_authors.fullname: Marco Dos Santos
paper_authors.type: user
paper_authors.user: dsantosmarco
paper_authors._id: 652de6c2d2937a53ed729bff
paper_authors.name: Stephen McAleer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:48:29.173Z
paper_authors._id: 64a84b1b4a7028998354d5d0
paper_authors.avatarUrl: /avatars/ea235fd1649b3446daf12dbed9b3a9db.svg
paper_authors.fullname: Stephen McAleer
paper_authors.type: user
paper_authors.user: mcaleste
paper_authors._id: 652de6c2d2937a53ed729c00
paper_authors.name: Albert Q. Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:48:36.012Z
paper_authors._id: 6137dff56ecbbda5c86755e9
paper_authors.avatarUrl: /avatars/1aadefd3acb7ad24df15463dadceb7d4.svg
paper_authors.fullname: Albert Q Jiang
paper_authors.type: user
paper_authors.user: aqj213
paper_authors._id: 652de6c2d2937a53ed729c01
paper_authors.name: Jia Deng
paper_authors._id: 652de6c2d2937a53ed729c02
paper_authors.name: Stella Biderman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T07:48:51.905Z
paper_authors._id: 60347d3660e3dd96631c9093
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
paper_authors.fullname: Stella Biderman
paper_authors.type: user
paper_authors.user: stellaathena
paper_authors._id: 652de6c2d2937a53ed729c03
paper_authors.name: Sean Welleck
paper_title: Llemma: An Open Language Model For Mathematics
paper_summary: We present Llemma, a large language model for mathematics. We continue
  pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web
  data containing mathematics, and mathematical code, yielding Llemma. On the
  MATH benchmark Llemma outperforms all known open base models, as well as the
  unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is
  capable of tool use and formal theorem proving without any further finetuning.
  We openly release all artifacts, including 7 billion and 34 billion parameter
  models, the Proof-Pile-2, and code to replicate our experiments.
paper_ai_summary: Llemma, a large language model pretrained on mathematical data, outperforms existing models and demonstrates tool use and formal theorem proving capabilities.
paper_ai_keywords: large language model mathematics pretraining Proof-Pile-2 MATH benchmark tool use formal theorem proving equi-parameter basis