When humans absorb information from the world, we innately use multiple
senses, such as seeing a busy street and hearing the sounds of car engines.
Today, we’re introducing an approach that brings machines one step closer to
humans’ ability to learn simultaneously, holistically, and directly from many
different forms of information — without the need for explicit supervision (the
process of organizing and labeling raw data). We have built and are open-
sourcing ImageBind, the first AI model capable of binding information from six
modalities. The model learns a single embedding, or shared representation
space, not just for text, image/video, and audio, but also for sensors that record
depth (3D), thermal (infrared radiation), and inertial measurement units (IMU),
which calculate motion and position. ImageBind equips machines with a holistic
understanding that connects objects in a photo with how they will sound, their
3D shape, how warm or cold they are, and how they move.
ImageBind_ Holistic AI learning across six modalities