title: Object Recognition as Next Token Prediction
summary: We present an approach to pose object recognition as next token prediction.
  The idea is to apply a language decoder that auto-regressively predicts the
  text tokens from image embeddings to form labels. To ground this prediction
  process in auto-regression, we customize a non-causal attention mask for the
  decoder, incorporating two key features: modeling tokens from different labels
  to be independent, and treating image tokens as a prefix. This masking
  mechanism inspires an efficient method - one-shot sampling - to simultaneously
  sample tokens of multiple labels in parallel and rank generated labels by their
  probabilities during inference. To further enhance the efficiency, we propose a
  simple strategy to construct a compact decoder by simply discarding the
  intermediate blocks of a pretrained language model. This approach yields a
  decoder that matches the full model's performance while being notably more
  efficient. The code is available at https://github.com/kaiyuyue/nxtp
paper_authors._id: 656e9a231f8ce3366d9eb166
paper_authors.name: Kaiyu Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T11:15:12.208Z
paper_authors._id: 640d0dbc8036cc2142273a83
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640d0dbc8036cc2142273a83/TQHyWWMGuXJKEM6fTPf-z.png
paper_authors.fullname: Kaiyu Yue
paper_authors.type: user
paper_authors.user: kaiyuyue
paper_authors._id: 656e9a231f8ce3366d9eb167
paper_authors.name: Bor-Chun Chen
paper_authors._id: 656e9a231f8ce3366d9eb168
paper_authors.name: Jonas Geiping
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T11:15:27.829Z
paper_authors._id: 63d86dbf3130cadcaf8bdd11
paper_authors.avatarUrl: /avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
paper_authors.fullname: Jonas Geiping
paper_authors.type: user
paper_authors.user: JonasGeiping
paper_authors._id: 656e9a231f8ce3366d9eb169
paper_authors.name: Hengduo Li
paper_authors._id: 656e9a231f8ce3366d9eb16a
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T11:15:48.070Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_authors._id: 656e9a231f8ce3366d9eb16b
paper_authors.name: Ser-Nam Lim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-26T10:38:01.652Z
paper_authors._id: 65884498ce38d143c435d279
paper_authors.avatarUrl: /avatars/96a917c40680a79a37dcb8fa75014c21.svg
paper_authors.fullname: Ser Nam Lim
paper_authors.type: user
paper_authors.user: sernam
paper_title: Object Recognition as Next Token Prediction
paper_summary: We present an approach to pose object recognition as next token prediction.
  The idea is to apply a language decoder that auto-regressively predicts the
  text tokens from image embeddings to form labels. To ground this prediction
  process in auto-regression, we customize a non-causal attention mask for the
  decoder, incorporating two key features: modeling tokens from different labels
  to be independent, and treating image tokens as a prefix. This masking
  mechanism inspires an efficient method - one-shot sampling - to simultaneously
  sample tokens of multiple labels in parallel and rank generated labels by their
  probabilities during inference. To further enhance the efficiency, we propose a
  simple strategy to construct a compact decoder by simply discarding the
  intermediate blocks of a pretrained language model. This approach yields a
  decoder that matches the full model's performance while being notably more
  efficient. The code is available at https://github.com/kaiyuyue/nxtp
paper_ai_summary: Pose object recognition is approached using next token prediction with customized non-causal attention masks and a compact language decoder, enhancing efficiency.
paper_ai_keywords: pose object recognition next token prediction language decoder auto-regressive prediction non-causal attention mask image embeddings one-shot sampling compact decoder pretrained language model