ProteinBench: A Holistic Evaluation of Protein Foundation Models
Recent years have witnessed a surge in the development of protein foundation
models, significantly improving performance in protein prediction and
generative tasks ranging from 3D structure prediction and protein design to
conformational dynamics. However, the capabilities and limitations associated
with these models remain poorly understood due to the absence of a unified
evaluation framework. To fill this gap, we introduce ProteinBench, a holistic
evaluation framework designed to enhance the transparency of protein foundation
models. Our approach consists of three key components: (i) A taxonomic
classification of tasks that broadly encompass the main challenges in the
protein domain, based on the relationships between different protein
modalities; (ii) A multi-metric evaluation approach that assesses performance
across four key dimensions: quality, novelty, diversity, and robustness; and
(iii) In-depth analyses from various user objectives, providing a holistic view
of model performance. Our comprehensive evaluation of protein foundation models
reveals several key findings that shed light on their current capabilities and
limitations. To promote transparency and facilitate further research, we
release the evaluation dataset, code, and a public leaderboard publicly for
further analysis and a general modular toolkit. We intend for ProteinBench to
be a living benchmark for establishing a standardized, in-depth evaluation
framework for protein foundation models, driving their development and
application while fostering collaboration within the field.
66e25fb578814a1b456c0dcf
Fei Ye
66e25fb578814a1b456c0dd0
Zaixiang Zheng
claimed_verified
2024-10-21T12:31:00.347Z
6347faa36d7f1d10f5ecdc18
/avatars/0fea067fa7bc8c4a797268b68a868756.svg
zzheng
user
zzheng
66e25fb578814a1b456c0dd1
Dongyu Xue
66e25fb578814a1b456c0dd2
Yuning Shen
66e25fb578814a1b456c0dd3
Lihao Wang
66e25fb578814a1b456c0dd4
Yiming Ma
66e25fb578814a1b456c0dd5
Yan Wang
66e25fb578814a1b456c0dd6
Xinyou Wang
claimed_verified
2024-10-22T08:03:25.704Z
64d0938b37a8b7adaf8c8219
/avatars/d8ea6a7fe938db2280b828aaeb41dcaf.svg
wang
user
airkingbd
66e25fb578814a1b456c0dd7
Xiangxin Zhou
admin_assigned
2024-09-12T09:06:05.131Z
66129c7b50350afe76757262
/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg
Xiangxin Zhou
user
zhouxiangxin
66e25fb578814a1b456c0dd8
Quanquan Gu
admin_assigned
2024-09-12T09:05:46.890Z
64c039128e2612254356bba5
/avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
Quanquan Gu
user
thughost
ProteinBench: A Holistic Evaluation of Protein Foundation Models
Recent years have witnessed a surge in the development of protein foundation
models, significantly improving performance in protein prediction and
generative tasks ranging from 3D structure prediction and protein design to
conformational dynamics. However, the capabilities and limitations associated
with these models remain poorly understood due to the absence of a unified
evaluation framework. To fill this gap, we introduce ProteinBench, a holistic
evaluation framework designed to enhance the transparency of protein foundation
models. Our approach consists of three key components: (i) A taxonomic
classification of tasks that broadly encompass the main challenges in the
protein domain, based on the relationships between different protein
modalities; (ii) A multi-metric evaluation approach that assesses performance
across four key dimensions: quality, novelty, diversity, and robustness; and
(iii) In-depth analyses from various user objectives, providing a holistic view
of model performance. Our comprehensive evaluation of protein foundation models
reveals several key findings that shed light on their current capabilities and
limitations. To promote transparency and facilitate further research, we
release the evaluation dataset, code, and a public leaderboard publicly for
further analysis and a general modular toolkit. We intend for ProteinBench to
be a living benchmark for establishing a standardized, in-depth evaluation
framework for protein foundation models, driving their development and
application while fostering collaboration within the field.
ProteinBench is an evaluation framework that enhances transparency in protein foundation models through taxonomic classification and multi-metric assessments.
protein foundation models 3D structure prediction protein design conformational dynamics ProteinBench taxonomic classification multi-metric evaluation quality novelty diversity robustness user objectives evaluation dataset public leaderboard modular toolkit