VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality
  Models
We present VLMEvalKit: an open-source toolkit for evaluating large
multi-modality models based on PyTorch. The toolkit aims to provide a
user-friendly and comprehensive framework for researchers and developers to
evaluate existing multi-modality models and publish reproducible evaluation
results. In VLMEvalKit, we implement over 70 different large multi-modality
models, including both proprietary APIs and open-source models, as well as more
than 20 different multi-modal benchmarks. By implementing a single interface,
new models can be easily added to the toolkit, while the toolkit automatically
handles the remaining workloads, including data preparation, distributed
inference, prediction post-processing, and metric calculation. Although the
toolkit is currently mainly used for evaluating large vision-language models,
its design is compatible with future updates that incorporate additional
modalities, such as audio and video. Based on the evaluation results obtained
with the toolkit, we host OpenVLM Leaderboard, a comprehensive leaderboard to
track the progress of multi-modality learning research. The toolkit is released
at https://github.com/open-compass/VLMEvalKit and is actively maintained.
66976e27883745077daa0df9
Haodong Duan
claimed_verified
2024-09-12T07:13:39.439Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
66976e27883745077daa0dfa
Junming Yang
66976e27883745077daa0dfb
Yuxuan Qiao
66976e27883745077daa0dfc
Xinyu Fang
claimed_verified
2024-11-22T10:02:12.022Z
64f5f8dd9b17cd59c453c57f
https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
Xinyu Fang
user
nebulae09
66976e27883745077daa0dfd
Lin Chen
claimed_verified
2024-07-18T09:06:36.632Z
64b02ec0e5000ae8a572ced5
https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
Lin Chen
user
Lin-Chen
66976e27883745077daa0dfe
Yuan Liu
claimed_verified
2024-09-23T06:58:43.584Z
64d47a7a508a6313e33faedd
https://cdn-avatars.huggingface.co/v1/production/uploads/64d47a7a508a6313e33faedd/8DrbXdxA-sKZb4jgNhvY1.jpeg
Yuan Liu
user
YuanLiuuuuuu
66976e27883745077daa0dff
Xiaoyi Dong
66976e27883745077daa0e00
Yuhang Zang
claimed_verified
2025-02-10T09:51:47.234Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
66976e27883745077daa0e01
Pan Zhang
66976e27883745077daa0e02
Jiaqi Wang
claimed_verified
2024-12-11T09:23:16.663Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
66976e27883745077daa0e03
Dahua Lin
66976e27883745077daa0e04
Kai Chen
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality
  Models
We present VLMEvalKit: an open-source toolkit for evaluating large
multi-modality models based on PyTorch. The toolkit aims to provide a
user-friendly and comprehensive framework for researchers and developers to
evaluate existing multi-modality models and publish reproducible evaluation
results. In VLMEvalKit, we implement over 70 different large multi-modality
models, including both proprietary APIs and open-source models, as well as more
than 20 different multi-modal benchmarks. By implementing a single interface,
new models can be easily added to the toolkit, while the toolkit automatically
handles the remaining workloads, including data preparation, distributed
inference, prediction post-processing, and metric calculation. Although the
toolkit is currently mainly used for evaluating large vision-language models,
its design is compatible with future updates that incorporate additional
modalities, such as audio and video. Based on the evaluation results obtained
with the toolkit, we host OpenVLM Leaderboard, a comprehensive leaderboard to
track the progress of multi-modality learning research. The toolkit is released
at https://github.com/open-compass/VLMEvalKit and is actively maintained.
VLMEvalKit is a PyTorch-based toolkit for evaluating large multi-modality models with a focus on vision-language models, offering a comprehensive framework for reproducible results and a leaderboard for research tracking.
multi-modality models PyTorch multi-modal benchmarks distributed inference prediction post-processing metric calculation OpenVLM Leaderboard