title: 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
summary: Current multimodal and multitask foundation models like 4M or UnifiedIO show
  promising results, but in practice their out-of-the-box abilities to accept
  diverse inputs and perform diverse tasks are limited by the (usually rather
  small) number of modalities and tasks they are trained on. In this paper, we
  expand upon the capabilities of them by training a single model on tens of
  highly diverse modalities and by performing co-training on large-scale
  multimodal datasets and text corpora. This includes training on several
  semantic and geometric modalities, feature maps from recent state of the art
  models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM
  and 4DHumans, and a range of new modalities that allow for novel ways to
  interact with the model and steer the generation, for example image metadata or
  color palettes. A crucial step in this process is performing discrete
  tokenization on various modalities, whether they are image-like, neural network
  feature maps, vectors, structured data like instance segmentation or human
  poses, or data that can be represented as text. Through this, we expand on the
  out-of-the-box capabilities of multimodal models and specifically show the
  possibility of training one model to solve at least 3x more tasks/modalities
  than existing ones and doing so without a loss in performance. This enables
  more fine-grained and controllable multimodal generation capabilities and
  allows us to study the distillation of models trained on diverse data and
  objectives into a unified model. We successfully scale the training to a three
  billion parameter model using tens of modalities and different datasets. The
  resulting models and training code are open sourced at 4m.epfl.ch.
paper_authors._id: 666bfd1c435a489ffd357149
paper_authors.name: Roman Bachmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-15T09:15:34.870Z
paper_authors._id: 624dc3c5953e603497e3a28c
paper_authors.avatarUrl: /avatars/955e0269f7e5fe3dbefeafa803a06d76.svg
paper_authors.fullname: Roman Bachmann
paper_authors.type: user
paper_authors.user: roman-bachmann
paper_authors._id: 666bfd1c435a489ffd35714a
paper_authors.name: Oğuzhan Fatih Kar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-15T09:15:45.099Z
paper_authors._id: 6303d05dd14428368d1866fd
paper_authors.avatarUrl: /avatars/6c86567fac3cb253e0595aa678a2af03.svg
paper_authors.fullname: Oguzhan Fatih Kar
paper_authors.type: user
paper_authors.user: ozo
paper_authors._id: 666bfd1c435a489ffd35714b
paper_authors.name: David Mizrahi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-15T09:15:51.911Z
paper_authors._id: 6257dd6bcf42812891aae42e
paper_authors.avatarUrl: /avatars/915d8c63e91486d8f8e48377314879db.svg
paper_authors.fullname: David Mizrahi
paper_authors.type: user
paper_authors.user: dmizr
paper_authors._id: 666bfd1c435a489ffd35714c
paper_authors.name: Ali Garjani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T13:28:46.922Z
paper_authors._id: 6381296d84022715e0d7708c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669409123069-noauth.png
paper_authors.fullname: Ali
paper_authors.type: user
paper_authors.user: aligniii
paper_authors._id: 666bfd1c435a489ffd35714d
paper_authors.name: Mingfei Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-15T09:16:01.127Z
paper_authors._id: 64388d5f1efe72ba48071c52
paper_authors.avatarUrl: /avatars/2653fba13e71f471e9d55c134fe25efc.svg
paper_authors.fullname: Mingfei Gao
paper_authors.type: user
paper_authors.user: fly6464
paper_authors._id: 666bfd1c435a489ffd35714e
paper_authors.name: David Griffiths
paper_authors._id: 666bfd1c435a489ffd35714f
paper_authors.name: Jiaming Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-15T09:16:52.055Z
paper_authors._id: 6432d9824521083b9d274693
paper_authors.avatarUrl: /avatars/34f1eb20b72aca78ac4d775ec18858f3.svg
paper_authors.fullname: Jiaming Hu
paper_authors.type: user
paper_authors.user: Masterkey
paper_authors._id: 666bfd1c435a489ffd357150
paper_authors.name: Afshin Dehghan
paper_authors._id: 666bfd1c435a489ffd357151
paper_authors.name: Amir Zamir
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-15T09:17:01.738Z
paper_authors._id: 666c7e0d29b9f769d6b2ee68
paper_authors.avatarUrl: /avatars/8261e3a8e6fdd2012a2e3f91279df110.svg
paper_authors.fullname: Amir Zamir
paper_authors.type: user
paper_authors.user: amir32002
paper_title: 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
paper_summary: Current multimodal and multitask foundation models like 4M or UnifiedIO show
  promising results, but in practice their out-of-the-box abilities to accept
  diverse inputs and perform diverse tasks are limited by the (usually rather
  small) number of modalities and tasks they are trained on. In this paper, we
  expand upon the capabilities of them by training a single model on tens of
  highly diverse modalities and by performing co-training on large-scale
  multimodal datasets and text corpora. This includes training on several
  semantic and geometric modalities, feature maps from recent state of the art
  models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM
  and 4DHumans, and a range of new modalities that allow for novel ways to
  interact with the model and steer the generation, for example image metadata or
  color palettes. A crucial step in this process is performing discrete
  tokenization on various modalities, whether they are image-like, neural network
  feature maps, vectors, structured data like instance segmentation or human
  poses, or data that can be represented as text. Through this, we expand on the
  out-of-the-box capabilities of multimodal models and specifically show the
  possibility of training one model to solve at least 3x more tasks/modalities
  than existing ones and doing so without a loss in performance. This enables
  more fine-grained and controllable multimodal generation capabilities and
  allows us to study the distillation of models trained on diverse data and
  objectives into a unified model. We successfully scale the training to a three
  billion parameter model using tens of modalities and different datasets. The
  resulting models and training code are open sourced at 4m.epfl.ch.
paper_ai_summary: A single multimodal model trained on a diverse set of modalities and tasks through co-learning and discrete tokenization achieves superior performance and flexibility compared to existing models.
paper_ai_keywords: discrete tokenization co-training UnifiedIO 4M DINOv2 ImageBind SAM 4DHumans multimodal multitask feature maps instance segmentation human poses