title: From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on
    Generalizability, Trustworthiness and Causality through Four Modalities
summary: Multi-modal Large Language Models (MLLMs) have shown impressive abilities in
  generating reasonable responses with respect to multi-modal contents. However,
  there is still a wide gap between the performance of recent MLLM-based
  applications and the expectation of the broad public, even though the most
  powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper
  strives to enhance understanding of the gap through the lens of a qualitative
  study on the generalizability, trustworthiness, and causal reasoning
  capabilities of recent proprietary and open-source MLLMs across four
  modalities: ie, text, code, image, and video, ultimately aiming to improve the
  transparency of MLLMs. We believe these properties are several representative
  factors that define the reliability of MLLMs, in supporting various downstream
  applications. To be specific, we evaluate the closed-source GPT-4 and Gemini
  and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed
  cases, where the qualitative results are then summarized into 12 scores (ie, 4
  modalities times 3 properties). In total, we uncover 14 empirical findings that
  are useful to understand the capabilities and limitations of both proprietary
  and open-source MLLMs, towards more reliable downstream multi-modal
  applications.
paper_authors._id: 65b70a43e2510e886d92d1c8
paper_authors.name: Chaochao Lu
paper_authors._id: 65b70a43e2510e886d92d1c9
paper_authors.name: Chen Qian
paper_authors._id: 65b70a43e2510e886d92d1ca
paper_authors.name: Guodong Zheng
paper_authors._id: 65b70a43e2510e886d92d1cb
paper_authors.name: Hongxing Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T09:39:09.407Z
paper_authors._id: 64a96a375a69e2ca889abdff
paper_authors.avatarUrl: /avatars/f288c66ace09d907f132a79a740a3701.svg
paper_authors.fullname: fanhongxing
paper_authors.type: user
paper_authors.user: fanhongxing
paper_authors._id: 65b70a43e2510e886d92d1cc
paper_authors.name: Hongzhi Gao
paper_authors._id: 65b70a43e2510e886d92d1cd
paper_authors.name: Jie Zhang
paper_authors._id: 65b70a43e2510e886d92d1ce
paper_authors.name: Jing Shao
paper_authors._id: 65b70a43e2510e886d92d1cf
paper_authors.name: Jingyi Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T17:15:33.312Z
paper_authors._id: 65b914e6b1e017206da47f7a
paper_authors.avatarUrl: /avatars/f5bcf66715dd8b3fa94fc4b424e3477a.svg
paper_authors.fullname: Jingyi Deng
paper_authors.type: user
paper_authors.user: JyiDeng
paper_authors._id: 65b70a43e2510e886d92d1d0
paper_authors.name: Jinlan Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T09:54:47.621Z
paper_authors._id: 618497ea8aaadc9253c2dfa9
paper_authors.avatarUrl: /avatars/2eb3954a99f5aede6f31b8ae49b8c910.svg
paper_authors.fullname: Fu Jinlan
paper_authors.type: user
paper_authors.user: Jinlan
paper_authors._id: 65b70a43e2510e886d92d1d1
paper_authors.name: Kexin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T09:57:06.761Z
paper_authors._id: 64ccd0104726a3f833823095
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ccd0104726a3f833823095/Q38tL8aD49BbcHACyvaea.jpeg
paper_authors.fullname: Kexin Huang
paper_authors.type: user
paper_authors.user: CaasiHUANG
paper_authors._id: 65b70a43e2510e886d92d1d2
paper_authors.name: Kunchang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T09:58:39.415Z
paper_authors._id: 61fb81006374891646732f37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
paper_authors.fullname: Kunchang Li
paper_authors.type: user
paper_authors.user: Andy1621
paper_authors._id: 65b70a43e2510e886d92d1d3
paper_authors.name: Lijun Li
paper_authors._id: 65b70a43e2510e886d92d1d4
paper_authors.name: Limin Wang
paper_authors._id: 65b70a43e2510e886d92d1d5
paper_authors.name: Lu Sheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-22T22:28:17.747Z
paper_authors._id: 65b722dbe02a17f0f8d1cc6b
paper_authors.avatarUrl: /avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
paper_authors.fullname: Lu Sheng
paper_authors.type: user
paper_authors.user: lsheng2024
paper_authors._id: 65b70a43e2510e886d92d1d6
paper_authors.name: Meiqi Chen
paper_authors._id: 65b70a43e2510e886d92d1d7
paper_authors.name: Ming Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T08:24:31.975Z
paper_authors._id: 65b71c0582d38451342f7334
paper_authors.avatarUrl: /avatars/e00248fb99604fa8f4f33f50584d376d.svg
paper_authors.fullname: Ming Zhang
paper_authors.type: user
paper_authors.user: konglongge
paper_authors._id: 65b70a43e2510e886d92d1d8
paper_authors.name: Qibing Ren
paper_authors._id: 65b70a43e2510e886d92d1d9
paper_authors.name: Sirui Chen
paper_authors._id: 65b70a43e2510e886d92d1da
paper_authors.name: Tao Gui
paper_authors._id: 65b70a43e2510e886d92d1db
paper_authors.name: Wanli Ouyang
paper_authors._id: 65b70a43e2510e886d92d1dc
paper_authors.name: Yali Wang
paper_authors._id: 65b70a43e2510e886d92d1dd
paper_authors.name: Yan Teng
paper_authors._id: 65b70a43e2510e886d92d1de
paper_authors.name: Yaru Wang
paper_authors._id: 65b70a43e2510e886d92d1df
paper_authors.name: Yi Wang
paper_authors._id: 65b70a43e2510e886d92d1e0
paper_authors.name: Yinan He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-18T08:37:27.410Z
paper_authors._id: 62aafa49f29ff279b51f0182
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
paper_authors.fullname: yinanhe
paper_authors.type: user
paper_authors.user: ynhe
paper_authors._id: 65b70a43e2510e886d92d1e1
paper_authors.name: Yingchun Wang
paper_authors._id: 65b70a43e2510e886d92d1e2
paper_authors.name: Yixu Wang
paper_authors._id: 65b70a43e2510e886d92d1e3
paper_authors.name: Yongting Zhang
paper_authors._id: 65b70a43e2510e886d92d1e4
paper_authors.name: Yu Qiao
paper_authors._id: 65b70a43e2510e886d92d1e5
paper_authors.name: Yujiong Shen
paper_authors._id: 65b70a43e2510e886d92d1e6
paper_authors.name: Yurong Mou
paper_authors._id: 65b70a43e2510e886d92d1e7
paper_authors.name: Yuxi Chen
paper_authors._id: 65b70a43e2510e886d92d1e8
paper_authors.name: Zaibin Zhang
paper_authors._id: 65b70a43e2510e886d92d1e9
paper_authors.name: Zhelun Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T09:40:35.221Z
paper_authors._id: 64f5d8b05619fe1fe06ed1a7
paper_authors.avatarUrl: /avatars/61016a0533760dafacefe8ce6e344318.svg
paper_authors.fullname: Zhelun Shi
paper_authors.type: user
paper_authors.user: CoachXP
paper_authors._id: 65b70a43e2510e886d92d1ea
paper_authors.name: Zhenfei Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T08:24:34.338Z
paper_authors._id: 64e314ad24809d7fa0f20fbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bHE0w_hjDFvU-Aul0_E7g.jpeg
paper_authors.fullname: Zhenfei Yin
paper_authors.type: user
paper_authors.user: JeremyYin
paper_authors._id: 65b70a43e2510e886d92d1eb
paper_authors.name: Zhipin Wang
paper_title: From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on
    Generalizability, Trustworthiness and Causality through Four Modalities
paper_summary: Multi-modal Large Language Models (MLLMs) have shown impressive abilities in
  generating reasonable responses with respect to multi-modal contents. However,
  there is still a wide gap between the performance of recent MLLM-based
  applications and the expectation of the broad public, even though the most
  powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper
  strives to enhance understanding of the gap through the lens of a qualitative
  study on the generalizability, trustworthiness, and causal reasoning
  capabilities of recent proprietary and open-source MLLMs across four
  modalities: ie, text, code, image, and video, ultimately aiming to improve the
  transparency of MLLMs. We believe these properties are several representative
  factors that define the reliability of MLLMs, in supporting various downstream
  applications. To be specific, we evaluate the closed-source GPT-4 and Gemini
  and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed
  cases, where the qualitative results are then summarized into 12 scores (ie, 4
  modalities times 3 properties). In total, we uncover 14 empirical findings that
  are useful to understand the capabilities and limitations of both proprietary
  and open-source MLLMs, towards more reliable downstream multi-modal
  applications.
paper_ai_summary: A qualitative study evaluates the generalizability, trustworthiness, and causal reasoning capabilities of MLLMs across text, code, image, and video modalities to understand their reliability for downstream applications.
paper_ai_keywords: Multi-modal Large Language Models MLLMs generalizability trustworthiness causal reasoning OpenAI's GPT-4 Google's Gemini qualitative study closed-source open-source empirical findings