PerceptionLM: Open-Access Data and Models for Detailed Visual
  Understanding
Vision-language models are integral to computer vision research, yet many
high-performing models remain closed-source, obscuring their data, design and
training recipe. The research community has responded by using distillation
from black-box models to label training data, achieving strong benchmark
results, at the cost of measurable scientific progress. However, without
knowing the details of the teacher model and its data sources, scientific
progress remains difficult to measure. In this paper, we study building a
Perception Language Model (PLM) in a fully open and reproducible framework for
transparent research in image and video understanding. We analyze standard
training pipelines without distillation from proprietary models and explore
large-scale synthetic data to identify critical data gaps, particularly in
detailed video understanding. To bridge these gaps, we release 2.8M
human-labeled instances of fine-grained video question-answer pairs and
spatio-temporally grounded video captions. Additionally, we introduce
PLM-VideoBench, a suite for evaluating challenging video understanding tasks
focusing on the ability to reason about "what", "where", "when", and "how" of a
video. We make our work fully reproducible by providing data, training recipes,
code & models.
6802c455f2384edf1c8ee068
Jang Hyun Cho
claimed_verified
2025-04-20T15:02:20.575Z
647a4ae6c6077fb66fbd0036
/avatars/ee08791366311da83bd5395012f3b1c2.svg
Jang Hyun (Vincent) Cho
user
janghyuncho7
6802c455f2384edf1c8ee069
Andrea Madotto
6802c455f2384edf1c8ee06a
Effrosyni Mavroudi
6802c455f2384edf1c8ee06b
Triantafyllos Afouras
6802c455f2384edf1c8ee06c
Tushar Nagarajan
6802c455f2384edf1c8ee06d
Muhammad Maaz
claimed_verified
2025-04-20T15:02:25.631Z
64807585856901b0edb8d68b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kmCo6brXhQ7SyxVx3lpsx.jpeg
Muhammad Maaz
user
mmaaz60
6802c455f2384edf1c8ee06e
Yale Song
6802c455f2384edf1c8ee06f
Tengyu Ma
6802c455f2384edf1c8ee070
Shuming Hu
claimed_verified
2025-04-20T15:02:23.100Z
6563c6846ef2a1d0f989f11b
/avatars/b03f6680ac3e718a1c38fec59a211b62.svg
Shuming Hu
user
shumingh
6802c455f2384edf1c8ee071
Suyog Jain
6802c455f2384edf1c8ee072
Miguel Martin
6802c455f2384edf1c8ee073
Huiyu Wang
6802c455f2384edf1c8ee074
Hanoona Rasheed
6802c455f2384edf1c8ee075
Peize Sun
claimed_verified
2025-04-20T15:02:17.940Z
640dc9bf8512ec51d7f0ac1a
https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
peizesun
user
peizesun
6802c455f2384edf1c8ee076
Po-Yao Huang
6802c455f2384edf1c8ee077
Daniel Bolya
6802c455f2384edf1c8ee078
Nikhila Ravi
6802c455f2384edf1c8ee079
Shashank Jain
6802c455f2384edf1c8ee07a
Tammy Stark
6802c455f2384edf1c8ee07b
Shane Moon
6802c455f2384edf1c8ee07c
Babak Damavandi
6802c455f2384edf1c8ee07d
Vivian Lee
6802c455f2384edf1c8ee07e
Andrew Westbury
6802c455f2384edf1c8ee07f
Salman Khan
6802c455f2384edf1c8ee080
Philipp Krähenbühl
6802c455f2384edf1c8ee081
Piotr Dollár
6802c455f2384edf1c8ee082
Lorenzo Torresani
6802c455f2384edf1c8ee083
Kristen Grauman
6802c455f2384edf1c8ee084
Christoph Feichtenhofer
PerceptionLM: Open-Access Data and Models for Detailed Visual
  Understanding
Vision-language models are integral to computer vision research, yet many
high-performing models remain closed-source, obscuring their data, design and
training recipe. The research community has responded by using distillation
from black-box models to label training data, achieving strong benchmark
results, at the cost of measurable scientific progress. However, without
knowing the details of the teacher model and its data sources, scientific
progress remains difficult to measure. In this paper, we study building a
Perception Language Model (PLM) in a fully open and reproducible framework for
transparent research in image and video understanding. We analyze standard
training pipelines without distillation from proprietary models and explore
large-scale synthetic data to identify critical data gaps, particularly in
detailed video understanding. To bridge these gaps, we release 2.8M
human-labeled instances of fine-grained video question-answer pairs and
spatio-temporally grounded video captions. Additionally, we introduce
PLM-VideoBench, a suite for evaluating challenging video understanding tasks
focusing on the ability to reason about "what", "where", "when", and "how" of a
video. We make our work fully reproducible by providing data, training recipes,
code & models.
A fully transparent Perception Language Model (PLM) for image and video understanding is developed without relying on proprietary models, using large-scale synthetic and human-labeled data and introducing PLM-VideoBench for evaluation.
Perception Language Model PLM video question-answer pairs spatio-temporally grounded video captions PLM-VideoBench video understanding