LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
Large language models (LLMs) have shown promise in proving formal theorems
using proof assistants such as Lean. However, existing methods are difficult to
reproduce or build on, due to private code, data, and large compute
requirements. This has created substantial barriers to research on machine
learning methods for theorem proving. This paper removes these barriers by
introducing LeanDojo: an open-source Lean playground consisting of toolkits,
data, models, and benchmarks. LeanDojo extracts data from Lean and enables
interaction with the proof environment programmatically. It contains
fine-grained annotations of premises in proofs, providing valuable data for
premise selection: a key bottleneck in theorem proving. Using this data, we
develop ReProver (Retrieval-Augmented Prover): the first LLM-based prover that
is augmented with retrieval for selecting premises from a vast math library. It
is inexpensive and needs only one GPU week of training. Our retriever leverages
LeanDojo's program analysis capability to identify accessible premises and hard
negative examples, which makes retrieval much more effective. Furthermore, we
construct a new benchmark consisting of 96,962 theorems and proofs extracted
from Lean's math library. It features challenging data split requiring the
prover to generalize to theorems relying on novel premises that are never used
in training. We use this benchmark for training and evaluation, and
experimental results demonstrate the effectiveness of ReProver over
non-retrieval baselines and GPT-4. We thus provide the first set of open-source
LLM-based theorem provers without any proprietary datasets and release it under
a permissive MIT license to facilitate further research.
649b8e7f9fc875a86329f76b
Kaiyu Yang
admin_assigned
2023-06-28T07:46:56.863Z
642a6a2816d4d8293c9c0246
/avatars/c2263d8227418e85845ddc0e22b0fd4e.svg
Kaiyu Yang
user
kaiyuy
649b8e7f9fc875a86329f76c
Aidan M. Swope
649b8e7f9fc875a86329f76d
Alex Gu
claimed_verified
2023-06-28T20:15:45.744Z
6179abf8ec6ce4dc2e5f2376
/avatars/d10c6a1b350146b36949a24220471295.svg
Alex Gu
user
minimario
649b8e7f9fc875a86329f76e
Rahul Chalamala
admin_assigned
2023-06-28T07:47:33.702Z
648abf5f0c2a2d8592ca07e7
/avatars/333a96e0b2e9420b2abacf538c776c18.svg
Rahul Chalamala
user
rchalamala
649b8e7f9fc875a86329f76f
Peiyang Song
claimed_verified
2023-06-28T20:15:48.235Z
649c5cf5c1ae48cf4d7dda34
/avatars/a2264945f9f876b690017a93f225f937.svg
Peiyang Song
user
p-song1
649b8e7f9fc875a86329f770
Shixing Yu
649b8e7f9fc875a86329f771
Saad Godil
649b8e7f9fc875a86329f772
Ryan Prenger
649b8e7f9fc875a86329f773
Anima Anandkumar
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
Large language models (LLMs) have shown promise in proving formal theorems
using proof assistants such as Lean. However, existing methods are difficult to
reproduce or build on, due to private code, data, and large compute
requirements. This has created substantial barriers to research on machine
learning methods for theorem proving. This paper removes these barriers by
introducing LeanDojo: an open-source Lean playground consisting of toolkits,
data, models, and benchmarks. LeanDojo extracts data from Lean and enables
interaction with the proof environment programmatically. It contains
fine-grained annotations of premises in proofs, providing valuable data for
premise selection: a key bottleneck in theorem proving. Using this data, we
develop ReProver (Retrieval-Augmented Prover): the first LLM-based prover that
is augmented with retrieval for selecting premises from a vast math library. It
is inexpensive and needs only one GPU week of training. Our retriever leverages
LeanDojo's program analysis capability to identify accessible premises and hard
negative examples, which makes retrieval much more effective. Furthermore, we
construct a new benchmark consisting of 96,962 theorems and proofs extracted
from Lean's math library. It features challenging data split requiring the
prover to generalize to theorems relying on novel premises that are never used
in training. We use this benchmark for training and evaluation, and
experimental results demonstrate the effectiveness of ReProver over
non-retrieval baselines and GPT-4. We thus provide the first set of open-source
LLM-based theorem provers without any proprietary datasets and release it under
a permissive MIT license to facilitate further research.
LeanDojo provides open-source tools, data, and benchmarks to facilitate research on machine learning methods for theorem proving, introducing ReProver, an LLM-based prover that uses retrieval for premise selection.
LeanDojo toolkits data models benchmarks proof assistants Lean premise selection ReProver retrieval-augmented prover program analysis hard negative examples theorems proofs GPT-4