title: PerceptionLM: Open-Access Data and Models for Detailed Visual
    Understanding
summary: Vision-language models are integral to computer vision research, yet many
  high-performing models remain closed-source, obscuring their data, design and
  training recipe. The research community has responded by using distillation
  from black-box models to label training data, achieving strong benchmark
  results, at the cost of measurable scientific progress. However, without
  knowing the details of the teacher model and its data sources, scientific
  progress remains difficult to measure. In this paper, we study building a
  Perception Language Model (PLM) in a fully open and reproducible framework for
  transparent research in image and video understanding. We analyze standard
  training pipelines without distillation from proprietary models and explore
  large-scale synthetic data to identify critical data gaps, particularly in
  detailed video understanding. To bridge these gaps, we release 2.8M
  human-labeled instances of fine-grained video question-answer pairs and
  spatio-temporally grounded video captions. Additionally, we introduce
  PLM-VideoBench, a suite for evaluating challenging video understanding tasks
  focusing on the ability to reason about "what", "where", "when", and "how" of a
  video. We make our work fully reproducible by providing data, training recipes,
  code & models.
paper_authors._id: 6802c455f2384edf1c8ee068
paper_authors.name: Jang Hyun Cho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:02:20.575Z
paper_authors._id: 647a4ae6c6077fb66fbd0036
paper_authors.avatarUrl: /avatars/ee08791366311da83bd5395012f3b1c2.svg
paper_authors.fullname: Jang Hyun (Vincent) Cho
paper_authors.type: user
paper_authors.user: janghyuncho7
paper_authors._id: 6802c455f2384edf1c8ee069
paper_authors.name: Andrea Madotto
paper_authors._id: 6802c455f2384edf1c8ee06a
paper_authors.name: Effrosyni Mavroudi
paper_authors._id: 6802c455f2384edf1c8ee06b
paper_authors.name: Triantafyllos Afouras
paper_authors._id: 6802c455f2384edf1c8ee06c
paper_authors.name: Tushar Nagarajan
paper_authors._id: 6802c455f2384edf1c8ee06d
paper_authors.name: Muhammad Maaz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:02:25.631Z
paper_authors._id: 64807585856901b0edb8d68b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kmCo6brXhQ7SyxVx3lpsx.jpeg
paper_authors.fullname: Muhammad Maaz
paper_authors.type: user
paper_authors.user: mmaaz60
paper_authors._id: 6802c455f2384edf1c8ee06e
paper_authors.name: Yale Song
paper_authors._id: 6802c455f2384edf1c8ee06f
paper_authors.name: Tengyu Ma
paper_authors._id: 6802c455f2384edf1c8ee070
paper_authors.name: Shuming Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:02:23.100Z
paper_authors._id: 6563c6846ef2a1d0f989f11b
paper_authors.avatarUrl: /avatars/b03f6680ac3e718a1c38fec59a211b62.svg
paper_authors.fullname: Shuming Hu
paper_authors.type: user
paper_authors.user: shumingh
paper_authors._id: 6802c455f2384edf1c8ee071
paper_authors.name: Suyog Jain
paper_authors._id: 6802c455f2384edf1c8ee072
paper_authors.name: Miguel Martin
paper_authors._id: 6802c455f2384edf1c8ee073
paper_authors.name: Huiyu Wang
paper_authors._id: 6802c455f2384edf1c8ee074
paper_authors.name: Hanoona Rasheed
paper_authors._id: 6802c455f2384edf1c8ee075
paper_authors.name: Peize Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:02:17.940Z
paper_authors._id: 640dc9bf8512ec51d7f0ac1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
paper_authors.fullname: peizesun
paper_authors.type: user
paper_authors.user: peizesun
paper_authors._id: 6802c455f2384edf1c8ee076
paper_authors.name: Po-Yao Huang
paper_authors._id: 6802c455f2384edf1c8ee077
paper_authors.name: Daniel Bolya
paper_authors._id: 6802c455f2384edf1c8ee078
paper_authors.name: Nikhila Ravi
paper_authors._id: 6802c455f2384edf1c8ee079
paper_authors.name: Shashank Jain
paper_authors._id: 6802c455f2384edf1c8ee07a
paper_authors.name: Tammy Stark
paper_authors._id: 6802c455f2384edf1c8ee07b
paper_authors.name: Shane Moon
paper_authors._id: 6802c455f2384edf1c8ee07c
paper_authors.name: Babak Damavandi
paper_authors._id: 6802c455f2384edf1c8ee07d
paper_authors.name: Vivian Lee
paper_authors._id: 6802c455f2384edf1c8ee07e
paper_authors.name: Andrew Westbury
paper_authors._id: 6802c455f2384edf1c8ee07f
paper_authors.name: Salman Khan
paper_authors._id: 6802c455f2384edf1c8ee080
paper_authors.name: Philipp Krähenbühl
paper_authors._id: 6802c455f2384edf1c8ee081
paper_authors.name: Piotr Dollár
paper_authors._id: 6802c455f2384edf1c8ee082
paper_authors.name: Lorenzo Torresani
paper_authors._id: 6802c455f2384edf1c8ee083
paper_authors.name: Kristen Grauman
paper_authors._id: 6802c455f2384edf1c8ee084
paper_authors.name: Christoph Feichtenhofer
paper_title: PerceptionLM: Open-Access Data and Models for Detailed Visual
    Understanding
paper_summary: Vision-language models are integral to computer vision research, yet many
  high-performing models remain closed-source, obscuring their data, design and
  training recipe. The research community has responded by using distillation
  from black-box models to label training data, achieving strong benchmark
  results, at the cost of measurable scientific progress. However, without
  knowing the details of the teacher model and its data sources, scientific
  progress remains difficult to measure. In this paper, we study building a
  Perception Language Model (PLM) in a fully open and reproducible framework for
  transparent research in image and video understanding. We analyze standard
  training pipelines without distillation from proprietary models and explore
  large-scale synthetic data to identify critical data gaps, particularly in
  detailed video understanding. To bridge these gaps, we release 2.8M
  human-labeled instances of fine-grained video question-answer pairs and
  spatio-temporally grounded video captions. Additionally, we introduce
  PLM-VideoBench, a suite for evaluating challenging video understanding tasks
  focusing on the ability to reason about "what", "where", "when", and "how" of a
  video. We make our work fully reproducible by providing data, training recipes,
  code & models.
paper_ai_summary: A fully transparent Perception Language Model (PLM) for image and video understanding is developed without relying on proprietary models, using large-scale synthetic and human-labeled data and introducing PLM-VideoBench for evaluation.
paper_ai_keywords: Perception Language Model PLM video question-answer pairs spatio-temporally grounded video captions PLM-VideoBench video understanding