Can MLLMs Understand the Deep Implication Behind Chinese Images?
As the capabilities of Multimodal Large Language Models (MLLMs) continue to
improve, the need for higher-order capability evaluation of MLLMs is
increasing. However, there is a lack of work evaluating MLLM for higher-order
perception and understanding of Chinese visual content. To fill the gap, we
introduce the **C**hinese **I**mage **I**mplication understanding
**Bench**mark, **CII-Bench**, which aims to assess the higher-order perception
and understanding capabilities of MLLMs for Chinese images. CII-Bench stands
out in several ways compared to existing benchmarks. Firstly, to ensure the
authenticity of the Chinese context, images in CII-Bench are sourced from the
Chinese Internet and manually reviewed, with corresponding answers also
manually crafted. Additionally, CII-Bench incorporates images that represent
Chinese traditional culture, such as famous Chinese traditional paintings,
which can deeply reflect the model's understanding of Chinese traditional
culture. Through extensive experiments on CII-Bench across multiple MLLMs, we
have made significant findings. Initially, a substantial gap is observed
between the performance of MLLMs and humans on CII-Bench. The highest accuracy
of MLLMs attains 64.4%, where as human accuracy averages 78.2%, peaking at an
impressive 81.0%. Subsequently, MLLMs perform worse on Chinese traditional
culture images, suggesting limitations in their ability to understand
high-level semantics and lack a deep knowledge base of Chinese traditional
culture. Finally, it is observed that most models exhibit enhanced accuracy
when image emotion hints are incorporated into the prompts. We believe that
CII-Bench will enable MLLMs to gain a better understanding of Chinese semantics
and Chinese-specific images, advancing the journey towards expert artificial
general intelligence (AGI). Our project is publicly available at
https://cii-bench.github.io/.
6712196cf92689ddb66f3434
Chenhao Zhang
claimed_verified
2024-10-18T08:25:08.521Z
647daf00cfca67bc50f9a99f
https://cdn-avatars.huggingface.co/v1/production/uploads/647daf00cfca67bc50f9a99f/8Snmk1V6lZ8ecdTW3POfm.jpeg
Chenhao(Leo) Zhang
user
MING-ZCH
6712196cf92689ddb66f3435
Xi Feng
6712196cf92689ddb66f3436
Yuelin Bai
6712196cf92689ddb66f3437
Xinrun Du
claimed_verified
2024-11-03T14:23:59.694Z
654907a4a1faff97850c4eff
/avatars/458c90151614bc7f116943b6e67d6b8a.svg
du
user
dododododo
6712196cf92689ddb66f3438
Jinchang Hou
6712196cf92689ddb66f3439
Kaixin Deng
claimed_verified
2024-10-24T07:15:54.642Z
6614973fe84c106921cc7dbe
/avatars/9e2fb698dbbbda2a1d95813d48b909e7.svg
den999
user
dkx2077
6712196cf92689ddb66f343a
Guangzeng Han
admin_assigned
2024-10-18T13:04:53.176Z
63337b0073c07e8aebb3df0d
/avatars/f0e55c3c61538d9521a28eda78cf31db.svg
GUANGZENG HAN
user
kwangju
6712196cf92689ddb66f343b
Qinrui Li
6712196cf92689ddb66f343c
Bingli Wang
admin_assigned
2024-10-18T13:05:06.585Z
658d0a228cff48d3a4612689
/avatars/70e297c6cb12d1bdde6d91c23f590b63.svg
Bingli Wang
user
BingliW
6712196cf92689ddb66f343d
Jiaheng Liu
admin_assigned
2024-10-18T13:05:12.954Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
6712196cf92689ddb66f343e
Xingwei Qu
6712196cf92689ddb66f343f
Yifei Zhang
extracted_confirmed
2024-10-30T06:09:35.226Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6712196cf92689ddb66f3440
Qixuan Zhao
admin_assigned
2024-10-18T13:05:26.660Z
6416fcaf249360df28d81237
/avatars/814d59b1e382b4b41c5334d95612483b.svg
Qixuan Zhao
user
MakiseKurisu
6712196cf92689ddb66f3441
Yiming Liang
admin_assigned
2024-10-18T13:08:40.195Z
6555e8d8a0c34cd61a6b9ce3
/avatars/71dc562cef4bd42f6b762f036357c800.svg
yimingliang
user
yimingliang
6712196cf92689ddb66f3442
Ziqiang Liu
admin_assigned
2024-10-18T13:06:53.846Z
61b9b5f9fd429dff1ed3bd58
https://cdn-avatars.huggingface.co/v1/production/uploads/61b9b5f9fd429dff1ed3bd58/GrfeNU4K1zts-gmQdGbVE.jpeg
Ziqiang Liu
user
icoderzqliu
6712196cf92689ddb66f3443
Feiteng Fang
admin_assigned
2024-10-18T13:07:05.165Z
634d4284f0a69955f66489f3
/avatars/9fac134537e6ef802f26d6d54c0e0bca.svg
feitengfang
user
feltoner
6712196cf92689ddb66f3444
Min Yang
6712196cf92689ddb66f3445
Wenhao Huang
6712196cf92689ddb66f3446
Chenghua Lin
admin_assigned
2024-10-18T13:06:18.628Z
6442ba123610a28a4ad5382f
/avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
Chenghua Lin
user
chenghualin
6712196cf92689ddb66f3447
Ge Zhang
admin_assigned
2024-10-18T13:06:24.181Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6712196cf92689ddb66f3448
Shiwen Ni
admin_assigned
2024-10-18T13:04:09.904Z
641944acf9d6f1d772ede902
/avatars/a2f88017f7ab0cbea6862201e35fe747.svg
ShiwenNi
user
ShiwenNi
Can MLLMs Understand the Deep Implication Behind Chinese Images?
As the capabilities of Multimodal Large Language Models (MLLMs) continue to
improve, the need for higher-order capability evaluation of MLLMs is
increasing. However, there is a lack of work evaluating MLLM for higher-order
perception and understanding of Chinese visual content. To fill the gap, we
introduce the **C**hinese **I**mage **I**mplication understanding
**Bench**mark, **CII-Bench**, which aims to assess the higher-order perception
and understanding capabilities of MLLMs for Chinese images. CII-Bench stands
out in several ways compared to existing benchmarks. Firstly, to ensure the
authenticity of the Chinese context, images in CII-Bench are sourced from the
Chinese Internet and manually reviewed, with corresponding answers also
manually crafted. Additionally, CII-Bench incorporates images that represent
Chinese traditional culture, such as famous Chinese traditional paintings,
which can deeply reflect the model's understanding of Chinese traditional
culture. Through extensive experiments on CII-Bench across multiple MLLMs, we
have made significant findings. Initially, a substantial gap is observed
between the performance of MLLMs and humans on CII-Bench. The highest accuracy
of MLLMs attains 64.4%, where as human accuracy averages 78.2%, peaking at an
impressive 81.0%. Subsequently, MLLMs perform worse on Chinese traditional
culture images, suggesting limitations in their ability to understand
high-level semantics and lack a deep knowledge base of Chinese traditional
culture. Finally, it is observed that most models exhibit enhanced accuracy
when image emotion hints are incorporated into the prompts. We believe that
CII-Bench will enable MLLMs to gain a better understanding of Chinese semantics
and Chinese-specific images, advancing the journey towards expert artificial
general intelligence (AGI). Our project is publicly available at
https://cii-bench.github.io/.
CII-Bench evaluates the higher-order perception and understanding of Chinese visual content by MLLMs, revealing limitations in handling Chinese traditional culture and suggesting improvements with emotion hints.
Multimodal Large Language Models MLLMs CII-Bench higher-order perception Chinese visual content Chinese traditional culture expert artificial general intelligence AGI