title: HEMM: Holistic Evaluation of Multimodal Foundation Models
summary: Multimodal foundation models that can holistically process text alongside
  images, video, audio, and other sensory modalities are increasingly used in a
  variety of real-world applications. However, it is challenging to characterize
  and study progress in multimodal foundation models, given the range of possible
  modeling decisions, tasks, and domains. In this paper, we introduce Holistic
  Evaluation of Multimodal Models (HEMM) to systematically evaluate the
  capabilities of multimodal foundation models across a set of 3 dimensions:
  basic skills, information flow, and real-world use cases. Basic multimodal
  skills are internal abilities required to solve problems, such as learning
  interactions across modalities, fine-grained alignment, multi-step reasoning,
  and the ability to handle external knowledge. Information flow studies how
  multimodal content changes during a task through querying, translation,
  editing, and fusion. Use cases span domain-specific challenges introduced in
  real-world multimedia, affective computing, natural sciences, healthcare, and
  human-computer interaction applications. Through comprehensive experiments
  across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e.g.,
  basic skills, information flows, and use cases) that pose challenges to today's
  models, and (2) distill performance trends regarding how different modeling
  dimensions (e.g., scale, pre-training data, multimodal alignment, pre-training,
  and instruction tuning objectives) influence performance. Our conclusions
  regarding challenging multimodal interactions, use cases, and tasks requiring
  reasoning and external knowledge, the benefits of data and model scale, and the
  impacts of instruction tuning yield actionable insights for future work in
  multimodal foundation models.
paper_authors._id: 668b8960ccbf926e742808b7
paper_authors.name: Paul Pu Liang
paper_authors._id: 668b8960ccbf926e742808b8
paper_authors.name: Akshay Goindani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T07:50:40.527Z
paper_authors._id: 649de1288f9890dcc25bdf19
paper_authors.avatarUrl: /avatars/caa7ccd06ccd49c5d14de317e21768d9.svg
paper_authors.fullname: Akshay Goindani
paper_authors.type: user
paper_authors.user: akshayg08
paper_authors._id: 668b8960ccbf926e742808b9
paper_authors.name: Talha Chafekar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T07:50:34.356Z
paper_authors._id: 6146f91079d11bc3c1714775
paper_authors.avatarUrl: /avatars/418247a90388a59f5d1ee8899144b329.svg
paper_authors.fullname: Talha Chafekar
paper_authors.type: user
paper_authors.user: talha1503
paper_authors._id: 668b8960ccbf926e742808ba
paper_authors.name: Leena Mathur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T07:50:27.878Z
paper_authors._id: 62bb76d32b4d8cb1f9710f63
paper_authors.avatarUrl: /avatars/ed695f5a39e1e6a2b2207ca0afad9eb9.svg
paper_authors.fullname: Leena Mathur
paper_authors.type: user
paper_authors.user: lmathur
paper_authors._id: 668b8960ccbf926e742808bb
paper_authors.name: Haofei Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T07:50:17.467Z
paper_authors._id: 636453547cf2c0b4f0a3ee1e
paper_authors.avatarUrl: /avatars/29b4bf0e6abd3b70bb0dbd58188e4ac8.svg
paper_authors.fullname: Haofei Yu
paper_authors.type: user
paper_authors.user: lwaekfjlk
paper_authors._id: 668b8960ccbf926e742808bc
paper_authors.name: Ruslan Salakhutdinov
paper_authors._id: 668b8960ccbf926e742808bd
paper_authors.name: Louis-Philippe Morency
paper_title: HEMM: Holistic Evaluation of Multimodal Foundation Models
paper_summary: Multimodal foundation models that can holistically process text alongside
  images, video, audio, and other sensory modalities are increasingly used in a
  variety of real-world applications. However, it is challenging to characterize
  and study progress in multimodal foundation models, given the range of possible
  modeling decisions, tasks, and domains. In this paper, we introduce Holistic
  Evaluation of Multimodal Models (HEMM) to systematically evaluate the
  capabilities of multimodal foundation models across a set of 3 dimensions:
  basic skills, information flow, and real-world use cases. Basic multimodal
  skills are internal abilities required to solve problems, such as learning
  interactions across modalities, fine-grained alignment, multi-step reasoning,
  and the ability to handle external knowledge. Information flow studies how
  multimodal content changes during a task through querying, translation,
  editing, and fusion. Use cases span domain-specific challenges introduced in
  real-world multimedia, affective computing, natural sciences, healthcare, and
  human-computer interaction applications. Through comprehensive experiments
  across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e.g.,
  basic skills, information flows, and use cases) that pose challenges to today's
  models, and (2) distill performance trends regarding how different modeling
  dimensions (e.g., scale, pre-training data, multimodal alignment, pre-training,
  and instruction tuning objectives) influence performance. Our conclusions
  regarding challenging multimodal interactions, use cases, and tasks requiring
  reasoning and external knowledge, the benefits of data and model scale, and the
  impacts of instruction tuning yield actionable insights for future work in
  multimodal foundation models.
paper_ai_keywords: multimodal foundation models holistic processing multimodal content modeling decisions systematic evaluation Basic Skills information flow real-world use cases interactions across modalities fine-grained alignment multi-step reasoning external knowledge querying translation editing fusion domain-specific challenges multimedia affective computing natural sciences healthcare human-computer interaction comprehensive experiments performance trends data and model scale pre-training data multimodal alignment pre-training instruction tuning objectives multimodal interactions reasoning external knowledge instruction tuning