title: Unified Model for Image, Video, Audio and Language Tasks
summary: Large Language Models (LLMs) have made the ambitious quest for generalist
  agents significantly far from being a fantasy. A key hurdle for building such
  general models is the diversity and heterogeneity of tasks and modalities. A
  promising solution is unification, allowing the support of a myriad of tasks
  and modalities within one unified framework. While few large models (e.g.,
  Flamingo (Alayrac et al., 2022), trained on massive datasets, can support more
  than two modalities, current small to mid-scale unified models are still
  limited to 2 modalities, usually image-text or video-text. The question that we
  ask is: is it possible to build efficiently a unified model that can support
  all modalities? To answer this, we propose UnIVAL, a step further towards this
  ambitious goal. Without relying on fancy datasets sizes or models with billions
  of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities
  and unifies text, images, video, and audio into a single model. Our model is
  efficiently pretrained on many tasks, based on task balancing and multimodal
  curriculum learning. UnIVAL shows competitive performance to existing
  state-of-the-art approaches, across image and video-text tasks. The feature
  representations learned from image and video-text modalities, allows the model
  to achieve competitive performance when finetuned on audio-text tasks, despite
  not being pretrained on audio. Thanks to the unified model, we propose a novel
  study on multimodal model merging via weight interpolation of models trained on
  different multimodal tasks, showing their benefits in particular for
  out-of-distribution generalization. Finally, we motivate unification by showing
  the synergy between tasks. The model weights and code are released here:
  https://github.com/mshukor/UnIVAL.
paper_authors._id: 64c867ecd11ed5b1e2df7441
paper_authors.name: Mustafa Shukor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T10:00:42.255Z
paper_authors._id: 62bdeedd01dc22b4d22a371e
paper_authors.avatarUrl: /avatars/6adc904fb1e08661d293a966270afabb.svg
paper_authors.fullname: Mustafa Shukor
paper_authors.type: user
paper_authors.user: mshukor
paper_authors._id: 64c867ecd11ed5b1e2df7442
paper_authors.name: Corentin Dancette
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T10:00:57.915Z
paper_authors._id: 62cdea59a9be5c195561c2b8
paper_authors.avatarUrl: /avatars/959b702e57718b9029634cb41772dcef.svg
paper_authors.fullname: Corentin Dancette
paper_authors.type: user
paper_authors.user: cdancette
paper_authors._id: 64c867ecd11ed5b1e2df7443
paper_authors.name: Alexandre Rame
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T10:01:18.163Z
paper_authors._id: 63c94ede00104ea998de19a6
paper_authors.avatarUrl: /avatars/273959d87f0c67747588cf0700d64039.svg
paper_authors.fullname: Alexandre Rame
paper_authors.type: user
paper_authors.user: alexrame
paper_authors._id: 64c867ecd11ed5b1e2df7444
paper_authors.name: Matthieu Cord
paper_title: Unified Model for Image, Video, Audio and Language Tasks
paper_summary: Large Language Models (LLMs) have made the ambitious quest for generalist
  agents significantly far from being a fantasy. A key hurdle for building such
  general models is the diversity and heterogeneity of tasks and modalities. A
  promising solution is unification, allowing the support of a myriad of tasks
  and modalities within one unified framework. While few large models (e.g.,
  Flamingo (Alayrac et al., 2022), trained on massive datasets, can support more
  than two modalities, current small to mid-scale unified models are still
  limited to 2 modalities, usually image-text or video-text. The question that we
  ask is: is it possible to build efficiently a unified model that can support
  all modalities? To answer this, we propose UnIVAL, a step further towards this
  ambitious goal. Without relying on fancy datasets sizes or models with billions
  of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities
  and unifies text, images, video, and audio into a single model. Our model is
  efficiently pretrained on many tasks, based on task balancing and multimodal
  curriculum learning. UnIVAL shows competitive performance to existing
  state-of-the-art approaches, across image and video-text tasks. The feature
  representations learned from image and video-text modalities, allows the model
  to achieve competitive performance when finetuned on audio-text tasks, despite
  not being pretrained on audio. Thanks to the unified model, we propose a novel
  study on multimodal model merging via weight interpolation of models trained on
  different multimodal tasks, showing their benefits in particular for
  out-of-distribution generalization. Finally, we motivate unification by showing
  the synergy between tasks. The model weights and code are released here:
  https://github.com/mshukor/UnIVAL.
paper_ai_summary: UnIVAL, a unified model with 0.25B parameters, efficiently supports text, images, video, and audio, demonstrating competitive performance and benefiting out-of-distribution generalization through multimodal weight interpolation.
paper_ai_keywords: Large Language Models generalist agents unification modalities Flamingo UnIVAL pretrained efficient multimodal curriculum learning feature representations weight interpolation out-of-distribution generalization