title: VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality
    Models
summary: We present VLMEvalKit: an open-source toolkit for evaluating large
  multi-modality models based on PyTorch. The toolkit aims to provide a
  user-friendly and comprehensive framework for researchers and developers to
  evaluate existing multi-modality models and publish reproducible evaluation
  results. In VLMEvalKit, we implement over 70 different large multi-modality
  models, including both proprietary APIs and open-source models, as well as more
  than 20 different multi-modal benchmarks. By implementing a single interface,
  new models can be easily added to the toolkit, while the toolkit automatically
  handles the remaining workloads, including data preparation, distributed
  inference, prediction post-processing, and metric calculation. Although the
  toolkit is currently mainly used for evaluating large vision-language models,
  its design is compatible with future updates that incorporate additional
  modalities, such as audio and video. Based on the evaluation results obtained
  with the toolkit, we host OpenVLM Leaderboard, a comprehensive leaderboard to
  track the progress of multi-modality learning research. The toolkit is released
  at https://github.com/open-compass/VLMEvalKit and is actively maintained.
paper_authors._id: 66976e27883745077daa0df9
paper_authors.name: Haodong Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-12T07:13:39.439Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 66976e27883745077daa0dfa
paper_authors.name: Junming Yang
paper_authors._id: 66976e27883745077daa0dfb
paper_authors.name: Yuxuan Qiao
paper_authors._id: 66976e27883745077daa0dfc
paper_authors.name: Xinyu Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:02:12.022Z
paper_authors._id: 64f5f8dd9b17cd59c453c57f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
paper_authors.fullname: Xinyu Fang
paper_authors.type: user
paper_authors.user: nebulae09
paper_authors._id: 66976e27883745077daa0dfd
paper_authors.name: Lin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:06:36.632Z
paper_authors._id: 64b02ec0e5000ae8a572ced5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
paper_authors.fullname: Lin Chen
paper_authors.type: user
paper_authors.user: Lin-Chen
paper_authors._id: 66976e27883745077daa0dfe
paper_authors.name: Yuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-23T06:58:43.584Z
paper_authors._id: 64d47a7a508a6313e33faedd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d47a7a508a6313e33faedd/8DrbXdxA-sKZb4jgNhvY1.jpeg
paper_authors.fullname: Yuan Liu
paper_authors.type: user
paper_authors.user: YuanLiuuuuuu
paper_authors._id: 66976e27883745077daa0dff
paper_authors.name: Xiaoyi Dong
paper_authors._id: 66976e27883745077daa0e00
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:51:47.234Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 66976e27883745077daa0e01
paper_authors.name: Pan Zhang
paper_authors._id: 66976e27883745077daa0e02
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:23:16.663Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_authors._id: 66976e27883745077daa0e03
paper_authors.name: Dahua Lin
paper_authors._id: 66976e27883745077daa0e04
paper_authors.name: Kai Chen
paper_title: VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality
    Models
paper_summary: We present VLMEvalKit: an open-source toolkit for evaluating large
  multi-modality models based on PyTorch. The toolkit aims to provide a
  user-friendly and comprehensive framework for researchers and developers to
  evaluate existing multi-modality models and publish reproducible evaluation
  results. In VLMEvalKit, we implement over 70 different large multi-modality
  models, including both proprietary APIs and open-source models, as well as more
  than 20 different multi-modal benchmarks. By implementing a single interface,
  new models can be easily added to the toolkit, while the toolkit automatically
  handles the remaining workloads, including data preparation, distributed
  inference, prediction post-processing, and metric calculation. Although the
  toolkit is currently mainly used for evaluating large vision-language models,
  its design is compatible with future updates that incorporate additional
  modalities, such as audio and video. Based on the evaluation results obtained
  with the toolkit, we host OpenVLM Leaderboard, a comprehensive leaderboard to
  track the progress of multi-modality learning research. The toolkit is released
  at https://github.com/open-compass/VLMEvalKit and is actively maintained.
paper_ai_summary: VLMEvalKit is a PyTorch-based toolkit for evaluating large multi-modality models with a focus on vision-language models, offering a comprehensive framework for reproducible results and a leaderboard for research tracking.
paper_ai_keywords: multi-modality models PyTorch multi-modal benchmarks distributed inference prediction post-processing metric calculation OpenVLM Leaderboard