Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal
  Language Model
Multimodal language models (MLLMs) are increasingly being implemented in
real-world environments, necessitating their ability to interpret 3D spaces and
comprehend temporal dynamics. Despite their potential, current top models
within our community still fall short in adequately understanding spatial and
temporal dimensions. We introduce Coarse Correspondence, a simple,
training-free, effective, and general-purpose visual prompting method to elicit
3D and temporal understanding in multimodal LLMs. Our method uses a lightweight
tracking model to find object correspondences between frames in a video or
between sets of image viewpoints. It selects the most frequent object instances
and visualizes them with markers with unique IDs in the image. With this simple
approach, we achieve state-of-the-art results on 3D understanding benchmarks
including ScanQA (+20.5\%) and a subset of OpenEQA (+9.7\%), and on long-form
video benchmarks such as EgoSchema (+6.0\%). We also curate a small diagnostic
dataset to evaluate whether MLLMs can reason about space from a described
viewpoint other than the camera viewpoint. Again, Coarse Correspondence
improves spatial perspective-taking abilities but we highlight that MLLMs
struggle with this task. Together, we demonstrate that our simple prompting
method can significantly aid downstream tasks that require 3D or temporal
reasoning.
66ac44e71cfb12c0874dff45
Benlin Liu
claimed_verified
2024-08-02T12:23:22.635Z
6326723c21c98f83d6ad128e
/avatars/6c208ff815c037d0a6a2ada3198e7fd4.svg
Benlin Liu
user
Tim666
66ac44e71cfb12c0874dff46
Yuhao Dong
admin_assigned
2024-08-02T07:41:48.696Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
66ac44e71cfb12c0874dff47
Yiqin Wang
admin_assigned
2024-08-02T07:41:58.025Z
63e065ae42591dda0b9b74cc
https://cdn-avatars.huggingface.co/v1/production/uploads/63e065ae42591dda0b9b74cc/U1zhG8saFcEo8bNewxugP.jpeg
Yiqin Wang
user
Yiqin
66ac44e71cfb12c0874dff48
Yongming Rao
admin_assigned
2024-08-02T07:42:04.286Z
63e4865354f51ea342d45d78
/avatars/2e7eccc878751331ca8b282f53e38899.svg
Yongming Rao
user
raoyongming
66ac44e71cfb12c0874dff49
Yansong Tang
66ac44e71cfb12c0874dff4a
Wei-Chiu Ma
admin_assigned
2024-08-02T07:42:18.825Z
624fb0c8841b0a7f1b583e72
/avatars/a695a528817f57a98c946056d6f9b2f9.svg
Wei-Chiu Ma
user
weichium
66ac44e71cfb12c0874dff4b
Ranjay Krishna
admin_assigned
2024-08-02T07:42:25.260Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal
  Language Model
Multimodal language models (MLLMs) are increasingly being implemented in
real-world environments, necessitating their ability to interpret 3D spaces and
comprehend temporal dynamics. Despite their potential, current top models
within our community still fall short in adequately understanding spatial and
temporal dimensions. We introduce Coarse Correspondence, a simple,
training-free, effective, and general-purpose visual prompting method to elicit
3D and temporal understanding in multimodal LLMs. Our method uses a lightweight
tracking model to find object correspondences between frames in a video or
between sets of image viewpoints. It selects the most frequent object instances
and visualizes them with markers with unique IDs in the image. With this simple
approach, we achieve state-of-the-art results on 3D understanding benchmarks
including ScanQA (+20.5\%) and a subset of OpenEQA (+9.7\%), and on long-form
video benchmarks such as EgoSchema (+6.0\%). We also curate a small diagnostic
dataset to evaluate whether MLLMs can reason about space from a described
viewpoint other than the camera viewpoint. Again, Coarse Correspondence
improves spatial perspective-taking abilities but we highlight that MLLMs
struggle with this task. Together, we demonstrate that our simple prompting
method can significantly aid downstream tasks that require 3D or temporal
reasoning.
A visual prompting method called Coarse Correspondence enhances 3D and temporal understanding in multimodal language models, achieving state-of-the-art results on relevant benchmarks.
multimodal language models 3D understanding temporal dynamics visual prompting object correspondences tracking model ScanQA OpenEQA EgoSchema spatial perspective-taking