Llemma: An Open Language Model For Mathematics
We present Llemma, a large language model for mathematics. We continue
pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web
data containing mathematics, and mathematical code, yielding Llemma. On the
MATH benchmark Llemma outperforms all known open base models, as well as the
unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is
capable of tool use and formal theorem proving without any further finetuning.
We openly release all artifacts, including 7 billion and 34 billion parameter
models, the Proof-Pile-2, and code to replicate our experiments.
652de6c2d2937a53ed729bfb
Zhangir Azerbayev
admin_assigned
2023-10-17T07:47:57.624Z
62cf683454477562dcbd8bf9
/avatars/404a8913f1bc5faa0e2b2ce767e0078e.svg
Zhangir Azerbayev
user
zhangirazerbayev
652de6c2d2937a53ed729bfc
Hailey Schoelkopf
admin_assigned
2023-10-17T07:48:06.895Z
62895a0215aeee85756062c4
https://cdn-avatars.huggingface.co/v1/production/uploads/1669665010552-62895a0215aeee85756062c4.jpeg
Hailey Schoelkopf
user
hails
652de6c2d2937a53ed729bfd
Keiran Paster
admin_assigned
2023-10-17T07:48:14.334Z
62d1cea65f9242437590d7e9
/avatars/8983669dbe58b1d9310843b34265b673.svg
Keiran Paster
user
keirp
652de6c2d2937a53ed729bfe
Marco Dos Santos
admin_assigned
2023-10-17T07:48:21.649Z
64146fc68ebec947b8c2fddc
https://cdn-avatars.huggingface.co/v1/production/uploads/64146fc68ebec947b8c2fddc/tTTdYhcg5cp2U8LGZustC.png
Marco Dos Santos
user
dsantosmarco
652de6c2d2937a53ed729bff
Stephen McAleer
admin_assigned
2023-10-17T07:48:29.173Z
64a84b1b4a7028998354d5d0
/avatars/ea235fd1649b3446daf12dbed9b3a9db.svg
Stephen McAleer
user
mcaleste
652de6c2d2937a53ed729c00
Albert Q. Jiang
admin_assigned
2023-10-17T07:48:36.012Z
6137dff56ecbbda5c86755e9
/avatars/1aadefd3acb7ad24df15463dadceb7d4.svg
Albert Q Jiang
user
aqj213
652de6c2d2937a53ed729c01
Jia Deng
652de6c2d2937a53ed729c02
Stella Biderman
admin_assigned
2023-10-17T07:48:51.905Z
60347d3660e3dd96631c9093
https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
Stella Biderman
user
stellaathena
652de6c2d2937a53ed729c03
Sean Welleck
Llemma: An Open Language Model For Mathematics
We present Llemma, a large language model for mathematics. We continue
pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web
data containing mathematics, and mathematical code, yielding Llemma. On the
MATH benchmark Llemma outperforms all known open base models, as well as the
unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is
capable of tool use and formal theorem proving without any further finetuning.
We openly release all artifacts, including 7 billion and 34 billion parameter
models, the Proof-Pile-2, and code to replicate our experiments.
Llemma, a large language model pretrained on mathematical data, outperforms existing models and demonstrates tool use and formal theorem proving capabilities.
large language model mathematics pretraining Proof-Pile-2 MATH benchmark tool use formal theorem proving equi-parameter basis