title: ProteinBench: A Holistic Evaluation of Protein Foundation Models
summary: Recent years have witnessed a surge in the development of protein foundation
  models, significantly improving performance in protein prediction and
  generative tasks ranging from 3D structure prediction and protein design to
  conformational dynamics. However, the capabilities and limitations associated
  with these models remain poorly understood due to the absence of a unified
  evaluation framework. To fill this gap, we introduce ProteinBench, a holistic
  evaluation framework designed to enhance the transparency of protein foundation
  models. Our approach consists of three key components: (i) A taxonomic
  classification of tasks that broadly encompass the main challenges in the
  protein domain, based on the relationships between different protein
  modalities; (ii) A multi-metric evaluation approach that assesses performance
  across four key dimensions: quality, novelty, diversity, and robustness; and
  (iii) In-depth analyses from various user objectives, providing a holistic view
  of model performance. Our comprehensive evaluation of protein foundation models
  reveals several key findings that shed light on their current capabilities and
  limitations. To promote transparency and facilitate further research, we
  release the evaluation dataset, code, and a public leaderboard publicly for
  further analysis and a general modular toolkit. We intend for ProteinBench to
  be a living benchmark for establishing a standardized, in-depth evaluation
  framework for protein foundation models, driving their development and
  application while fostering collaboration within the field.
paper_authors._id: 66e25fb578814a1b456c0dcf
paper_authors.name: Fei Ye
paper_authors._id: 66e25fb578814a1b456c0dd0
paper_authors.name: Zaixiang Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T12:31:00.347Z
paper_authors._id: 6347faa36d7f1d10f5ecdc18
paper_authors.avatarUrl: /avatars/0fea067fa7bc8c4a797268b68a868756.svg
paper_authors.fullname: zzheng
paper_authors.type: user
paper_authors.user: zzheng
paper_authors._id: 66e25fb578814a1b456c0dd1
paper_authors.name: Dongyu Xue
paper_authors._id: 66e25fb578814a1b456c0dd2
paper_authors.name: Yuning Shen
paper_authors._id: 66e25fb578814a1b456c0dd3
paper_authors.name: Lihao Wang
paper_authors._id: 66e25fb578814a1b456c0dd4
paper_authors.name: Yiming Ma
paper_authors._id: 66e25fb578814a1b456c0dd5
paper_authors.name: Yan Wang
paper_authors._id: 66e25fb578814a1b456c0dd6
paper_authors.name: Xinyou Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:03:25.704Z
paper_authors._id: 64d0938b37a8b7adaf8c8219
paper_authors.avatarUrl: /avatars/d8ea6a7fe938db2280b828aaeb41dcaf.svg
paper_authors.fullname: wang
paper_authors.type: user
paper_authors.user: airkingbd
paper_authors._id: 66e25fb578814a1b456c0dd7
paper_authors.name: Xiangxin Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T09:06:05.131Z
paper_authors._id: 66129c7b50350afe76757262
paper_authors.avatarUrl: /avatars/a2f4fac076b9d658a0d904ed54960f6f.svg
paper_authors.fullname: Xiangxin Zhou
paper_authors.type: user
paper_authors.user: zhouxiangxin
paper_authors._id: 66e25fb578814a1b456c0dd8
paper_authors.name: Quanquan Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T09:05:46.890Z
paper_authors._id: 64c039128e2612254356bba5
paper_authors.avatarUrl: /avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
paper_authors.fullname: Quanquan Gu
paper_authors.type: user
paper_authors.user: thughost
paper_title: ProteinBench: A Holistic Evaluation of Protein Foundation Models
paper_summary: Recent years have witnessed a surge in the development of protein foundation
  models, significantly improving performance in protein prediction and
  generative tasks ranging from 3D structure prediction and protein design to
  conformational dynamics. However, the capabilities and limitations associated
  with these models remain poorly understood due to the absence of a unified
  evaluation framework. To fill this gap, we introduce ProteinBench, a holistic
  evaluation framework designed to enhance the transparency of protein foundation
  models. Our approach consists of three key components: (i) A taxonomic
  classification of tasks that broadly encompass the main challenges in the
  protein domain, based on the relationships between different protein
  modalities; (ii) A multi-metric evaluation approach that assesses performance
  across four key dimensions: quality, novelty, diversity, and robustness; and
  (iii) In-depth analyses from various user objectives, providing a holistic view
  of model performance. Our comprehensive evaluation of protein foundation models
  reveals several key findings that shed light on their current capabilities and
  limitations. To promote transparency and facilitate further research, we
  release the evaluation dataset, code, and a public leaderboard publicly for
  further analysis and a general modular toolkit. We intend for ProteinBench to
  be a living benchmark for establishing a standardized, in-depth evaluation
  framework for protein foundation models, driving their development and
  application while fostering collaboration within the field.
paper_ai_summary: ProteinBench is an evaluation framework that enhances transparency in protein foundation models through taxonomic classification and multi-metric assessments.
paper_ai_keywords: protein foundation models 3D structure prediction protein design conformational dynamics ProteinBench taxonomic classification multi-metric evaluation quality novelty diversity robustness user objectives evaluation dataset public leaderboard modular toolkit