title: Perception Encoder: The best visual embeddings are not at the output of
    the network
summary: We introduce Perception Encoder (PE), a state-of-the-art encoder for image
  and video understanding trained via simple vision-language learning.
  Traditionally, vision encoders have relied on a variety of pretraining
  objectives, each tailored to specific downstream tasks such as classification,
  captioning, or localization. Surprisingly, after scaling our carefully tuned
  image pretraining recipe and refining with our robust video data engine, we
  find that contrastive vision-language training alone can produce strong,
  general embeddings for all of these downstream tasks. There is only one caveat:
  these embeddings are hidden within the intermediate layers of the network. To
  draw them out, we introduce two alignment methods, language alignment for
  multimodal language modeling, and spatial alignment for dense prediction.
  Together with the core contrastive checkpoint, our PE family of models achieves
  state-of-the-art performance on a wide variety of tasks, including zero-shot
  image and video classification and retrieval; document, image, and video Q&A;
  and spatial tasks such as detection, depth estimation, and tracking. To foster
  further research, we are releasing our models, code, and a novel dataset of
  synthetically and human-annotated videos.
paper_authors._id: 680237af2724430ec1defd6f
paper_authors.name: Daniel Bolya
paper_authors._id: 680237af2724430ec1defd70
paper_authors.name: Po-Yao Huang
paper_authors._id: 680237af2724430ec1defd71
paper_authors.name: Peize Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:02:49.858Z
paper_authors._id: 640dc9bf8512ec51d7f0ac1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
paper_authors.fullname: peizesun
paper_authors.type: user
paper_authors.user: peizesun
paper_authors._id: 680237af2724430ec1defd72
paper_authors.name: Jang Hyun Cho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:02:53.798Z
paper_authors._id: 647a4ae6c6077fb66fbd0036
paper_authors.avatarUrl: /avatars/ee08791366311da83bd5395012f3b1c2.svg
paper_authors.fullname: Jang Hyun (Vincent) Cho
paper_authors.type: user
paper_authors.user: janghyuncho7
paper_authors._id: 680237af2724430ec1defd73
paper_authors.name: Andrea Madotto
paper_authors._id: 680237af2724430ec1defd74
paper_authors.name: Chen Wei
paper_authors._id: 680237af2724430ec1defd75
paper_authors.name: Tengyu Ma
paper_authors._id: 680237af2724430ec1defd76
paper_authors.name: Jiale Zhi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:02:57.074Z
paper_authors._id: 64c9e07ef7f4ccb5ea6a5f8d
paper_authors.avatarUrl: /avatars/57e990d4dddd749c6cb50c3f0150a332.svg
paper_authors.fullname: Jiale Zhi
paper_authors.type: user
paper_authors.user: jz2023
paper_authors._id: 680237af2724430ec1defd77
paper_authors.name: Jathushan Rajasegaran
paper_authors._id: 680237af2724430ec1defd78
paper_authors.name: Hanoona Rasheed
paper_authors._id: 680237af2724430ec1defd79
paper_authors.name: Junke Wang
paper_authors._id: 680237af2724430ec1defd7a
paper_authors.name: Marco Monteiro
paper_authors._id: 680237af2724430ec1defd7b
paper_authors.name: Hu Xu
paper_authors._id: 680237af2724430ec1defd7c
paper_authors.name: Shiyu Dong
paper_authors._id: 680237af2724430ec1defd7d
paper_authors.name: Nikhila Ravi
paper_authors._id: 680237af2724430ec1defd7e
paper_authors.name: Daniel Li
paper_authors._id: 680237af2724430ec1defd7f
paper_authors.name: Piotr Dollár
paper_authors._id: 680237af2724430ec1defd80
paper_authors.name: Christoph Feichtenhofer
paper_title: Perception Encoder: The best visual embeddings are not at the output of
    the network
paper_summary: We introduce Perception Encoder (PE), a state-of-the-art encoder for image
  and video understanding trained via simple vision-language learning.
  Traditionally, vision encoders have relied on a variety of pretraining
  objectives, each tailored to specific downstream tasks such as classification,
  captioning, or localization. Surprisingly, after scaling our carefully tuned
  image pretraining recipe and refining with our robust video data engine, we
  find that contrastive vision-language training alone can produce strong,
  general embeddings for all of these downstream tasks. There is only one caveat:
  these embeddings are hidden within the intermediate layers of the network. To
  draw them out, we introduce two alignment methods, language alignment for
  multimodal language modeling, and spatial alignment for dense prediction.
  Together with the core contrastive checkpoint, our PE family of models achieves
  state-of-the-art performance on a wide variety of tasks, including zero-shot
  image and video classification and retrieval; document, image, and video Q&A;
  and spatial tasks such as detection, depth estimation, and tracking. To foster
  further research, we are releasing our models, code, and a novel dataset of
  synthetically and human-annotated videos.
paper_ai_summary: Perception Encoder, trained via contrastive vision-language learning, achieves state-of-the-art performance across various image and video tasks using intermediate embeddings extracted through alignment methods.
paper_ai_keywords: Perception Encoder contrastive vision-language training language alignment spatial alignment zero-shot classification zero-shot retrieval document Q&A image Q&A video Q&A detection depth estimation tracking