Perceptual Decoupling for Scalable Multi-modal Reasoning via Reward-Optimized Captioning
Paper
•
2506.04559
•
Published
•
2
•
1
Kai Chen
KaiChen1998
AI & ML interests
Omni-modal Large Language Models & Controllable Visual World Modeling & Autonomous Driving
Recent Activity
liked
a Space
2 days ago
Emova-ollm/RACRO-demo
posted
an
update
3 days ago
🤔 Advanced reasoning LLMs keep releasing before your current MLLM alignment is done? Try our RACRO! Train once, flexible change to novel LLM reasoners during inference time!
📢 RACRO is a novel methodology to build multi-modal large reasoning models. By decoupling multi-modal reasoning into 1) query-conditioned captioning and 2) text-only reasoning, we achieve SoTA results on multi-modal reasoning benchmarks, while supporting flexible changes to any advanced reasoning models during inference. We further propose CRO, a novel GRPO-variant to reinforce query-conditioned captioning with only verifiable data for multi-modal mathematical questions.
✨ Highlights
✅ State-of-the-art multi-modal reasoning: we achieve SoTA performance on multi-modal mathematical benchmarks, exceeding advanced commercial models like Claude-3.7-Sonnet and Gemini-2.0-Flash.
✅ Inference-time scalability: thanks to the perceptual decoupling, we can flexibly change LLM reasoners during inference, providing a unique inference-time scalability for multi-modal reasoning.
✅ Highly efficient: With only a single round of Caption Reward Optimization (CRO) training on ~39K samples, RACRO gets rid of burdensome multi-modal alignment (e.g., 4.1T tokens for Qwen2.5-VL).
🔥 You are all welcome to try and star!
- Paper: https://huggingface.co/papers/2506.04559
- Github: https://github.com/gyhdog99/RACRO2
- Demo: https://huggingface.co/spaces/Emova-ollm/RACRO-demo
updated
a Space
3 days ago
Emova-ollm/RACRO-demo
Organizations
KaiChen1998's activity
commented
a
paper
12 days ago
Improve language tag
1
#1 opened about 2 months ago
by
lbourdois
commented
5
papers
3 months ago
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Paper
•
2409.18042
•
Published
•
41
•
13
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Paper
•
2409.18042
•
Published
•
41
•
13
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Paper
•
2409.18042
•
Published
•
41
•
13
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Paper
•
2409.18042
•
Published
•
41
•
13
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Paper
•
2409.18042
•
Published
•
41
•
13
Adding `safetensors` variant of this model
#1 opened 5 months ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened 5 months ago
by
SFconvertbot
commented
a
paper
8 months ago
commented
a
paper
9 months ago