title: Can MLLMs Understand the Deep Implication Behind Chinese Images?
summary: As the capabilities of Multimodal Large Language Models (MLLMs) continue to
  improve, the need for higher-order capability evaluation of MLLMs is
  increasing. However, there is a lack of work evaluating MLLM for higher-order
  perception and understanding of Chinese visual content. To fill the gap, we
  introduce the **C**hinese **I**mage **I**mplication understanding
  **Bench**mark, **CII-Bench**, which aims to assess the higher-order perception
  and understanding capabilities of MLLMs for Chinese images. CII-Bench stands
  out in several ways compared to existing benchmarks. Firstly, to ensure the
  authenticity of the Chinese context, images in CII-Bench are sourced from the
  Chinese Internet and manually reviewed, with corresponding answers also
  manually crafted. Additionally, CII-Bench incorporates images that represent
  Chinese traditional culture, such as famous Chinese traditional paintings,
  which can deeply reflect the model's understanding of Chinese traditional
  culture. Through extensive experiments on CII-Bench across multiple MLLMs, we
  have made significant findings. Initially, a substantial gap is observed
  between the performance of MLLMs and humans on CII-Bench. The highest accuracy
  of MLLMs attains 64.4%, where as human accuracy averages 78.2%, peaking at an
  impressive 81.0%. Subsequently, MLLMs perform worse on Chinese traditional
  culture images, suggesting limitations in their ability to understand
  high-level semantics and lack a deep knowledge base of Chinese traditional
  culture. Finally, it is observed that most models exhibit enhanced accuracy
  when image emotion hints are incorporated into the prompts. We believe that
  CII-Bench will enable MLLMs to gain a better understanding of Chinese semantics
  and Chinese-specific images, advancing the journey towards expert artificial
  general intelligence (AGI). Our project is publicly available at
  https://cii-bench.github.io/.
paper_authors._id: 6712196cf92689ddb66f3434
paper_authors.name: Chenhao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:25:08.521Z
paper_authors._id: 647daf00cfca67bc50f9a99f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647daf00cfca67bc50f9a99f/8Snmk1V6lZ8ecdTW3POfm.jpeg
paper_authors.fullname: Chenhao(Leo) Zhang
paper_authors.type: user
paper_authors.user: MING-ZCH
paper_authors._id: 6712196cf92689ddb66f3435
paper_authors.name: Xi Feng
paper_authors._id: 6712196cf92689ddb66f3436
paper_authors.name: Yuelin Bai
paper_authors._id: 6712196cf92689ddb66f3437
paper_authors.name: Xinrun Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:23:59.694Z
paper_authors._id: 654907a4a1faff97850c4eff
paper_authors.avatarUrl: /avatars/458c90151614bc7f116943b6e67d6b8a.svg
paper_authors.fullname: du
paper_authors.type: user
paper_authors.user: dododododo
paper_authors._id: 6712196cf92689ddb66f3438
paper_authors.name: Jinchang Hou
paper_authors._id: 6712196cf92689ddb66f3439
paper_authors.name: Kaixin Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:15:54.642Z
paper_authors._id: 6614973fe84c106921cc7dbe
paper_authors.avatarUrl: /avatars/9e2fb698dbbbda2a1d95813d48b909e7.svg
paper_authors.fullname: den999
paper_authors.type: user
paper_authors.user: dkx2077
paper_authors._id: 6712196cf92689ddb66f343a
paper_authors.name: Guangzeng Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:04:53.176Z
paper_authors._id: 63337b0073c07e8aebb3df0d
paper_authors.avatarUrl: /avatars/f0e55c3c61538d9521a28eda78cf31db.svg
paper_authors.fullname: GUANGZENG HAN
paper_authors.type: user
paper_authors.user: kwangju
paper_authors._id: 6712196cf92689ddb66f343b
paper_authors.name: Qinrui Li
paper_authors._id: 6712196cf92689ddb66f343c
paper_authors.name: Bingli Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:05:06.585Z
paper_authors._id: 658d0a228cff48d3a4612689
paper_authors.avatarUrl: /avatars/70e297c6cb12d1bdde6d91c23f590b63.svg
paper_authors.fullname: Bingli Wang
paper_authors.type: user
paper_authors.user: BingliW
paper_authors._id: 6712196cf92689ddb66f343d
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:05:12.954Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 6712196cf92689ddb66f343e
paper_authors.name: Xingwei Qu
paper_authors._id: 6712196cf92689ddb66f343f
paper_authors.name: Yifei Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-30T06:09:35.226Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6712196cf92689ddb66f3440
paper_authors.name: Qixuan Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:05:26.660Z
paper_authors._id: 6416fcaf249360df28d81237
paper_authors.avatarUrl: /avatars/814d59b1e382b4b41c5334d95612483b.svg
paper_authors.fullname: Qixuan Zhao
paper_authors.type: user
paper_authors.user: MakiseKurisu
paper_authors._id: 6712196cf92689ddb66f3441
paper_authors.name: Yiming Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:08:40.195Z
paper_authors._id: 6555e8d8a0c34cd61a6b9ce3
paper_authors.avatarUrl: /avatars/71dc562cef4bd42f6b762f036357c800.svg
paper_authors.fullname: yimingliang
paper_authors.type: user
paper_authors.user: yimingliang
paper_authors._id: 6712196cf92689ddb66f3442
paper_authors.name: Ziqiang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:06:53.846Z
paper_authors._id: 61b9b5f9fd429dff1ed3bd58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61b9b5f9fd429dff1ed3bd58/GrfeNU4K1zts-gmQdGbVE.jpeg
paper_authors.fullname: Ziqiang Liu
paper_authors.type: user
paper_authors.user: icoderzqliu
paper_authors._id: 6712196cf92689ddb66f3443
paper_authors.name: Feiteng Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:07:05.165Z
paper_authors._id: 634d4284f0a69955f66489f3
paper_authors.avatarUrl: /avatars/9fac134537e6ef802f26d6d54c0e0bca.svg
paper_authors.fullname: feitengfang
paper_authors.type: user
paper_authors.user: feltoner
paper_authors._id: 6712196cf92689ddb66f3444
paper_authors.name: Min Yang
paper_authors._id: 6712196cf92689ddb66f3445
paper_authors.name: Wenhao Huang
paper_authors._id: 6712196cf92689ddb66f3446
paper_authors.name: Chenghua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:06:18.628Z
paper_authors._id: 6442ba123610a28a4ad5382f
paper_authors.avatarUrl: /avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
paper_authors.fullname: Chenghua Lin
paper_authors.type: user
paper_authors.user: chenghualin
paper_authors._id: 6712196cf92689ddb66f3447
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:06:24.181Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6712196cf92689ddb66f3448
paper_authors.name: Shiwen Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:04:09.904Z
paper_authors._id: 641944acf9d6f1d772ede902
paper_authors.avatarUrl: /avatars/a2f88017f7ab0cbea6862201e35fe747.svg
paper_authors.fullname: ShiwenNi
paper_authors.type: user
paper_authors.user: ShiwenNi
paper_title: Can MLLMs Understand the Deep Implication Behind Chinese Images?
paper_summary: As the capabilities of Multimodal Large Language Models (MLLMs) continue to
  improve, the need for higher-order capability evaluation of MLLMs is
  increasing. However, there is a lack of work evaluating MLLM for higher-order
  perception and understanding of Chinese visual content. To fill the gap, we
  introduce the **C**hinese **I**mage **I**mplication understanding
  **Bench**mark, **CII-Bench**, which aims to assess the higher-order perception
  and understanding capabilities of MLLMs for Chinese images. CII-Bench stands
  out in several ways compared to existing benchmarks. Firstly, to ensure the
  authenticity of the Chinese context, images in CII-Bench are sourced from the
  Chinese Internet and manually reviewed, with corresponding answers also
  manually crafted. Additionally, CII-Bench incorporates images that represent
  Chinese traditional culture, such as famous Chinese traditional paintings,
  which can deeply reflect the model's understanding of Chinese traditional
  culture. Through extensive experiments on CII-Bench across multiple MLLMs, we
  have made significant findings. Initially, a substantial gap is observed
  between the performance of MLLMs and humans on CII-Bench. The highest accuracy
  of MLLMs attains 64.4%, where as human accuracy averages 78.2%, peaking at an
  impressive 81.0%. Subsequently, MLLMs perform worse on Chinese traditional
  culture images, suggesting limitations in their ability to understand
  high-level semantics and lack a deep knowledge base of Chinese traditional
  culture. Finally, it is observed that most models exhibit enhanced accuracy
  when image emotion hints are incorporated into the prompts. We believe that
  CII-Bench will enable MLLMs to gain a better understanding of Chinese semantics
  and Chinese-specific images, advancing the journey towards expert artificial
  general intelligence (AGI). Our project is publicly available at
  https://cii-bench.github.io/.