4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
Current multimodal and multitask foundation models like 4M or UnifiedIO show
promising results, but in practice their out-of-the-box abilities to accept
diverse inputs and perform diverse tasks are limited by the (usually rather
small) number of modalities and tasks they are trained on. In this paper, we
expand upon the capabilities of them by training a single model on tens of
highly diverse modalities and by performing co-training on large-scale
multimodal datasets and text corpora. This includes training on several
semantic and geometric modalities, feature maps from recent state of the art
models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM
and 4DHumans, and a range of new modalities that allow for novel ways to
interact with the model and steer the generation, for example image metadata or
color palettes. A crucial step in this process is performing discrete
tokenization on various modalities, whether they are image-like, neural network
feature maps, vectors, structured data like instance segmentation or human
poses, or data that can be represented as text. Through this, we expand on the
out-of-the-box capabilities of multimodal models and specifically show the
possibility of training one model to solve at least 3x more tasks/modalities
than existing ones and doing so without a loss in performance. This enables
more fine-grained and controllable multimodal generation capabilities and
allows us to study the distillation of models trained on diverse data and
objectives into a unified model. We successfully scale the training to a three
billion parameter model using tens of modalities and different datasets. The
resulting models and training code are open sourced at 4m.epfl.ch.
666bfd1c435a489ffd357149
Roman Bachmann
admin_assigned
2024-06-15T09:15:34.870Z
624dc3c5953e603497e3a28c
/avatars/955e0269f7e5fe3dbefeafa803a06d76.svg
Roman Bachmann
user
roman-bachmann
666bfd1c435a489ffd35714a
Oğuzhan Fatih Kar
admin_assigned
2024-06-15T09:15:45.099Z
6303d05dd14428368d1866fd
/avatars/6c86567fac3cb253e0595aa678a2af03.svg
Oguzhan Fatih Kar
user
ozo
666bfd1c435a489ffd35714b
David Mizrahi
admin_assigned
2024-06-15T09:15:51.911Z
6257dd6bcf42812891aae42e
/avatars/915d8c63e91486d8f8e48377314879db.svg
David Mizrahi
user
dmizr
666bfd1c435a489ffd35714c
Ali Garjani
claimed_verified
2024-06-18T13:28:46.922Z
6381296d84022715e0d7708c
https://cdn-avatars.huggingface.co/v1/production/uploads/1669409123069-noauth.png
Ali
user
aligniii
666bfd1c435a489ffd35714d
Mingfei Gao
admin_assigned
2024-06-15T09:16:01.127Z
64388d5f1efe72ba48071c52
/avatars/2653fba13e71f471e9d55c134fe25efc.svg
Mingfei Gao
user
fly6464
666bfd1c435a489ffd35714e
David Griffiths
666bfd1c435a489ffd35714f
Jiaming Hu
admin_assigned
2024-06-15T09:16:52.055Z
6432d9824521083b9d274693
/avatars/34f1eb20b72aca78ac4d775ec18858f3.svg
Jiaming Hu
user
Masterkey
666bfd1c435a489ffd357150
Afshin Dehghan
666bfd1c435a489ffd357151
Amir Zamir
admin_assigned
2024-06-15T09:17:01.738Z
666c7e0d29b9f769d6b2ee68
/avatars/8261e3a8e6fdd2012a2e3f91279df110.svg
Amir Zamir
user
amir32002
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
Current multimodal and multitask foundation models like 4M or UnifiedIO show
promising results, but in practice their out-of-the-box abilities to accept
diverse inputs and perform diverse tasks are limited by the (usually rather
small) number of modalities and tasks they are trained on. In this paper, we
expand upon the capabilities of them by training a single model on tens of
highly diverse modalities and by performing co-training on large-scale
multimodal datasets and text corpora. This includes training on several
semantic and geometric modalities, feature maps from recent state of the art
models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM
and 4DHumans, and a range of new modalities that allow for novel ways to
interact with the model and steer the generation, for example image metadata or
color palettes. A crucial step in this process is performing discrete
tokenization on various modalities, whether they are image-like, neural network
feature maps, vectors, structured data like instance segmentation or human
poses, or data that can be represented as text. Through this, we expand on the
out-of-the-box capabilities of multimodal models and specifically show the
possibility of training one model to solve at least 3x more tasks/modalities
than existing ones and doing so without a loss in performance. This enables
more fine-grained and controllable multimodal generation capabilities and
allows us to study the distillation of models trained on diverse data and
objectives into a unified model. We successfully scale the training to a three
billion parameter model using tens of modalities and different datasets. The
resulting models and training code are open sourced at 4m.epfl.ch.
A single multimodal model trained on a diverse set of modalities and tasks through co-learning and discrete tokenization achieves superior performance and flexibility compared to existing models.
discrete tokenization co-training UnifiedIO 4M DINOv2 ImageBind SAM 4DHumans multimodal multitask feature maps instance segmentation human poses