Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
Multimodal language models (MLMs) still face challenges in fundamental visual
perception tasks where specialized models excel. Tasks requiring reasoning
about 3D structures benefit from depth estimation, and reasoning about 2D
object instances benefits from object detection. Yet, MLMs can not produce
intermediate depth or boxes to reason over. Finetuning MLMs on relevant data
doesn't generalize well and outsourcing computation to specialized vision tools
is too compute-intensive and memory-inefficient. To address this, we introduce
Perception Tokens, intrinsic image representations designed to assist reasoning
tasks where language is insufficient. Perception tokens act as auxiliary
reasoning tokens, akin to chain-of-thought prompts in language models. For
example, in a depth-related task, an MLM augmented with perception tokens can
reason by generating a depth map as tokens, enabling it to solve the problem
effectively. We propose AURORA, a training method that augments MLMs with
perception tokens for improved reasoning over visual inputs. AURORA leverages a
VQVAE to transform intermediate image representations, such as depth maps into
a tokenized format and bounding box tokens, which is then used in a multi-task
training framework. AURORA achieves notable improvements across counting
benchmarks: +10.8% on BLINK, +11.3% on CVBench, and +8.3% on SEED-Bench,
outperforming finetuning approaches in generalization across datasets. It also
improves on relative depth: over +6% on BLINK. With perception tokens, AURORA
expands the scope of MLMs beyond language-based reasoning, paving the way for
more effective visual reasoning capabilities.
675880b250e4b1a8f1f54e78
Mahtab Bigverdi
admin_assigned
2024-12-11T13:37:32.728Z
636d316effbe479c979692e8
/avatars/f1004a8d788c3ee94f2cfd1abf314485.svg
Mahtab Bigverdi
user
Mahtab
675880b250e4b1a8f1f54e79
Zelun Luo
675880b250e4b1a8f1f54e7a
Cheng-Yu Hsieh
admin_assigned
2024-12-11T13:38:32.254Z
6408e8663461c51cf73480e4
/avatars/7e4cbdb4b933181c673c9dc9ad041767.svg
Cheng-Yu Hsieh
user
cydhsieh01
675880b250e4b1a8f1f54e7b
Ethan Shen
admin_assigned
2024-12-11T13:38:09.784Z
65f7ee498fb2b15357cd6286
/avatars/a060fdd27b6a23346a8772d49df39018.svg
shen
user
Ethanshen
675880b250e4b1a8f1f54e7c
Dongping Chen
admin_assigned
2024-12-11T13:37:53.324Z
65e2be1e630e2db23829ee8d
/avatars/294f9ba909037f03669dc0bb80cabfe3.svg
Dongping Chen
user
fjchendp
675880b250e4b1a8f1f54e7d
Linda G. Shapiro
675880b250e4b1a8f1f54e7e
Ranjay Krishna
admin_assigned
2024-12-11T13:37:18.312Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
Multimodal language models (MLMs) still face challenges in fundamental visual
perception tasks where specialized models excel. Tasks requiring reasoning
about 3D structures benefit from depth estimation, and reasoning about 2D
object instances benefits from object detection. Yet, MLMs can not produce
intermediate depth or boxes to reason over. Finetuning MLMs on relevant data
doesn't generalize well and outsourcing computation to specialized vision tools
is too compute-intensive and memory-inefficient. To address this, we introduce
Perception Tokens, intrinsic image representations designed to assist reasoning
tasks where language is insufficient. Perception tokens act as auxiliary
reasoning tokens, akin to chain-of-thought prompts in language models. For
example, in a depth-related task, an MLM augmented with perception tokens can
reason by generating a depth map as tokens, enabling it to solve the problem
effectively. We propose AURORA, a training method that augments MLMs with
perception tokens for improved reasoning over visual inputs. AURORA leverages a
VQVAE to transform intermediate image representations, such as depth maps into
a tokenized format and bounding box tokens, which is then used in a multi-task
training framework. AURORA achieves notable improvements across counting
benchmarks: +10.8% on BLINK, +11.3% on CVBench, and +8.3% on SEED-Bench,
outperforming finetuning approaches in generalization across datasets. It also
improves on relative depth: over +6% on BLINK. With perception tokens, AURORA
expands the scope of MLMs beyond language-based reasoning, paving the way for
more effective visual reasoning capabilities.
AURORA, a method that incorporates Perception Tokens into MLMs, improves visual reasoning by transforming intermediate image representations into tokenized formats, enhancing performance on tasks like counting and depth estimation.
multimodal language models MLMs depth estimation object detection perception tokens chain-of-thought prompts VQVAE tokenized format bounding box tokens multi-task training BLINK CVBench SEED-Bench relative depth