title: Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
summary: Multimodal language models (MLMs) still face challenges in fundamental visual
  perception tasks where specialized models excel. Tasks requiring reasoning
  about 3D structures benefit from depth estimation, and reasoning about 2D
  object instances benefits from object detection. Yet, MLMs can not produce
  intermediate depth or boxes to reason over. Finetuning MLMs on relevant data
  doesn't generalize well and outsourcing computation to specialized vision tools
  is too compute-intensive and memory-inefficient. To address this, we introduce
  Perception Tokens, intrinsic image representations designed to assist reasoning
  tasks where language is insufficient. Perception tokens act as auxiliary
  reasoning tokens, akin to chain-of-thought prompts in language models. For
  example, in a depth-related task, an MLM augmented with perception tokens can
  reason by generating a depth map as tokens, enabling it to solve the problem
  effectively. We propose AURORA, a training method that augments MLMs with
  perception tokens for improved reasoning over visual inputs. AURORA leverages a
  VQVAE to transform intermediate image representations, such as depth maps into
  a tokenized format and bounding box tokens, which is then used in a multi-task
  training framework. AURORA achieves notable improvements across counting
  benchmarks: +10.8% on BLINK, +11.3% on CVBench, and +8.3% on SEED-Bench,
  outperforming finetuning approaches in generalization across datasets. It also
  improves on relative depth: over +6% on BLINK. With perception tokens, AURORA
  expands the scope of MLMs beyond language-based reasoning, paving the way for
  more effective visual reasoning capabilities.
paper_authors._id: 675880b250e4b1a8f1f54e78
paper_authors.name: Mahtab Bigverdi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:37:32.728Z
paper_authors._id: 636d316effbe479c979692e8
paper_authors.avatarUrl: /avatars/f1004a8d788c3ee94f2cfd1abf314485.svg
paper_authors.fullname: Mahtab Bigverdi
paper_authors.type: user
paper_authors.user: Mahtab
paper_authors._id: 675880b250e4b1a8f1f54e79
paper_authors.name: Zelun Luo
paper_authors._id: 675880b250e4b1a8f1f54e7a
paper_authors.name: Cheng-Yu Hsieh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:38:32.254Z
paper_authors._id: 6408e8663461c51cf73480e4
paper_authors.avatarUrl: /avatars/7e4cbdb4b933181c673c9dc9ad041767.svg
paper_authors.fullname: Cheng-Yu Hsieh
paper_authors.type: user
paper_authors.user: cydhsieh01
paper_authors._id: 675880b250e4b1a8f1f54e7b
paper_authors.name: Ethan Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:38:09.784Z
paper_authors._id: 65f7ee498fb2b15357cd6286
paper_authors.avatarUrl: /avatars/a060fdd27b6a23346a8772d49df39018.svg
paper_authors.fullname: shen
paper_authors.type: user
paper_authors.user: Ethanshen
paper_authors._id: 675880b250e4b1a8f1f54e7c
paper_authors.name: Dongping Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:37:53.324Z
paper_authors._id: 65e2be1e630e2db23829ee8d
paper_authors.avatarUrl: /avatars/294f9ba909037f03669dc0bb80cabfe3.svg
paper_authors.fullname: Dongping Chen
paper_authors.type: user
paper_authors.user: fjchendp
paper_authors._id: 675880b250e4b1a8f1f54e7d
paper_authors.name: Linda G. Shapiro
paper_authors._id: 675880b250e4b1a8f1f54e7e
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T13:37:18.312Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_title: Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
paper_summary: Multimodal language models (MLMs) still face challenges in fundamental visual
  perception tasks where specialized models excel. Tasks requiring reasoning
  about 3D structures benefit from depth estimation, and reasoning about 2D
  object instances benefits from object detection. Yet, MLMs can not produce
  intermediate depth or boxes to reason over. Finetuning MLMs on relevant data
  doesn't generalize well and outsourcing computation to specialized vision tools
  is too compute-intensive and memory-inefficient. To address this, we introduce
  Perception Tokens, intrinsic image representations designed to assist reasoning
  tasks where language is insufficient. Perception tokens act as auxiliary
  reasoning tokens, akin to chain-of-thought prompts in language models. For
  example, in a depth-related task, an MLM augmented with perception tokens can
  reason by generating a depth map as tokens, enabling it to solve the problem
  effectively. We propose AURORA, a training method that augments MLMs with
  perception tokens for improved reasoning over visual inputs. AURORA leverages a
  VQVAE to transform intermediate image representations, such as depth maps into
  a tokenized format and bounding box tokens, which is then used in a multi-task
  training framework. AURORA achieves notable improvements across counting
  benchmarks: +10.8% on BLINK, +11.3% on CVBench, and +8.3% on SEED-Bench,
  outperforming finetuning approaches in generalization across datasets. It also
  improves on relative depth: over +6% on BLINK. With perception tokens, AURORA
  expands the scope of MLMs beyond language-based reasoning, paving the way for
  more effective visual reasoning capabilities.
paper_ai_keywords: Perception Tokens intrinsic image representations auxiliary reasoning tokens chain-of-thought prompts depth map VQVAE tokenized format bounding box tokens multi-task training framework counting benchmarks BLINK CVBench SEED-Bench relative depth