title: Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal
    Language Model
summary: Multimodal language models (MLLMs) are increasingly being implemented in
  real-world environments, necessitating their ability to interpret 3D spaces and
  comprehend temporal dynamics. Despite their potential, current top models
  within our community still fall short in adequately understanding spatial and
  temporal dimensions. We introduce Coarse Correspondence, a simple,
  training-free, effective, and general-purpose visual prompting method to elicit
  3D and temporal understanding in multimodal LLMs. Our method uses a lightweight
  tracking model to find object correspondences between frames in a video or
  between sets of image viewpoints. It selects the most frequent object instances
  and visualizes them with markers with unique IDs in the image. With this simple
  approach, we achieve state-of-the-art results on 3D understanding benchmarks
  including ScanQA (+20.5\%) and a subset of OpenEQA (+9.7\%), and on long-form
  video benchmarks such as EgoSchema (+6.0\%). We also curate a small diagnostic
  dataset to evaluate whether MLLMs can reason about space from a described
  viewpoint other than the camera viewpoint. Again, Coarse Correspondence
  improves spatial perspective-taking abilities but we highlight that MLLMs
  struggle with this task. Together, we demonstrate that our simple prompting
  method can significantly aid downstream tasks that require 3D or temporal
  reasoning.
paper_authors._id: 66ac44e71cfb12c0874dff45
paper_authors.name: Benlin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-02T12:23:22.635Z
paper_authors._id: 6326723c21c98f83d6ad128e
paper_authors.avatarUrl: /avatars/6c208ff815c037d0a6a2ada3198e7fd4.svg
paper_authors.fullname: Benlin Liu
paper_authors.type: user
paper_authors.user: Tim666
paper_authors._id: 66ac44e71cfb12c0874dff46
paper_authors.name: Yuhao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T07:41:48.696Z
paper_authors._id: 652965773a416e1f2173443b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
paper_authors.fullname: Yuhao Dong
paper_authors.type: user
paper_authors.user: THUdyh
paper_authors._id: 66ac44e71cfb12c0874dff47
paper_authors.name: Yiqin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T07:41:58.025Z
paper_authors._id: 63e065ae42591dda0b9b74cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e065ae42591dda0b9b74cc/U1zhG8saFcEo8bNewxugP.jpeg
paper_authors.fullname: Yiqin Wang
paper_authors.type: user
paper_authors.user: Yiqin
paper_authors._id: 66ac44e71cfb12c0874dff48
paper_authors.name: Yongming Rao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T07:42:04.286Z
paper_authors._id: 63e4865354f51ea342d45d78
paper_authors.avatarUrl: /avatars/2e7eccc878751331ca8b282f53e38899.svg
paper_authors.fullname: Yongming Rao
paper_authors.type: user
paper_authors.user: raoyongming
paper_authors._id: 66ac44e71cfb12c0874dff49
paper_authors.name: Yansong Tang
paper_authors._id: 66ac44e71cfb12c0874dff4a
paper_authors.name: Wei-Chiu Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T07:42:18.825Z
paper_authors._id: 624fb0c8841b0a7f1b583e72
paper_authors.avatarUrl: /avatars/a695a528817f57a98c946056d6f9b2f9.svg
paper_authors.fullname: Wei-Chiu Ma
paper_authors.type: user
paper_authors.user: weichium
paper_authors._id: 66ac44e71cfb12c0874dff4b
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T07:42:25.260Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_title: Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal
    Language Model
paper_summary: Multimodal language models (MLLMs) are increasingly being implemented in
  real-world environments, necessitating their ability to interpret 3D spaces and
  comprehend temporal dynamics. Despite their potential, current top models
  within our community still fall short in adequately understanding spatial and
  temporal dimensions. We introduce Coarse Correspondence, a simple,
  training-free, effective, and general-purpose visual prompting method to elicit
  3D and temporal understanding in multimodal LLMs. Our method uses a lightweight
  tracking model to find object correspondences between frames in a video or
  between sets of image viewpoints. It selects the most frequent object instances
  and visualizes them with markers with unique IDs in the image. With this simple
  approach, we achieve state-of-the-art results on 3D understanding benchmarks
  including ScanQA (+20.5\%) and a subset of OpenEQA (+9.7\%), and on long-form
  video benchmarks such as EgoSchema (+6.0\%). We also curate a small diagnostic
  dataset to evaluate whether MLLMs can reason about space from a described
  viewpoint other than the camera viewpoint. Again, Coarse Correspondence
  improves spatial perspective-taking abilities but we highlight that MLLMs
  struggle with this task. Together, we demonstrate that our simple prompting
  method can significantly aid downstream tasks that require 3D or temporal
  reasoning.
paper_ai_keywords: Multimodal LLMs (MLLMs) Coarse Correspondence 3D spaces temporal dynamics parallel and temporal understanding lightweight tracking model object correspondences visual markers unique IDs ScanQA OpenEQA EgoSchema long-form video spatial perspective-taking downstream tasks 3D reasoning temporal reasoning