title: Adapting LLaMA Decoder to Vision Transformer
summary: This work examines whether decoder-only Transformers such as LLaMA, which
  were originally designed for large language models (LLMs), can be adapted to
  the computer vision field. We first "LLaMAfy" a standard ViT step-by-step to
  align with LLaMA's architecture, and find that directly applying a casual mask
  to the self-attention brings an attention collapse issue, resulting in the
  failure to the network training. We suggest to reposition the class token
  behind the image tokens with a post-sequence class token technique to overcome
  this challenge, enabling causal self-attention to efficiently capture the
  entire image's information. Additionally, we develop a soft mask strategy that
  gradually introduces a casual mask to the self-attention at the onset of
  training to facilitate the optimization behavior. The tailored model, dubbed as
  image LLaMA (iLLaMA), is akin to LLaMA in architecture and enables direct
  supervised learning. Its causal self-attention boosts computational efficiency
  and learns complex representation by elevating attention map ranks. iLLaMA
  rivals the performance with its encoder-only counterparts, achieving 75.1%
  ImageNet top-1 accuracy with only 5.7M parameters. Scaling the model to ~310M
  and pre-training on ImageNet-21K further enhances the accuracy to 86.0%.
  Extensive experiments demonstrate iLLaMA's reliable properties: calibration,
  shape-texture bias, quantization compatibility, ADE20K segmentation and CIFAR
  transfer learning. We hope our study can kindle fresh views to visual model
  design in the wave of LLMs. Pre-trained models and codes are available here.
paper_authors._id: 66173f994488ff8dfa933e33
paper_authors.name: Jiahao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T09:02:32.979Z
paper_authors._id: 65994517d3f2137415bb4d5b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65994517d3f2137415bb4d5b/PhjWetLTMn3bThWFLGCxR.png
paper_authors.fullname: WANG Jiahao
paper_authors.type: user
paper_authors.user: techmonsterwang
paper_authors._id: 66173f994488ff8dfa933e34
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:18:52.599Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 66173f994488ff8dfa933e35
paper_authors.name: Mengzhao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:19:09.508Z
paper_authors._id: 64aea082704210bf815e7551
paper_authors.avatarUrl: /avatars/5c8dc0df57596c526b2bccea21835f53.svg
paper_authors.fullname: Mengzhao Chen
paper_authors.type: user
paper_authors.user: ChenMnZ
paper_authors._id: 66173f994488ff8dfa933e36
paper_authors.name: Chengyue Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:19:24.943Z
paper_authors._id: 617526c9de8feb54b0ce45ad
paper_authors.avatarUrl: /avatars/7faf8c6f71fc318a0113d780d376c381.svg
paper_authors.fullname: Wu Chengyue
paper_authors.type: user
paper_authors.user: WuChengyue
paper_authors._id: 66173f994488ff8dfa933e37
paper_authors.name: Yong Liu
paper_authors._id: 66173f994488ff8dfa933e38
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:19:37.309Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_authors._id: 66173f994488ff8dfa933e39
paper_authors.name: Songyang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T14:20:17.538Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 66173f994488ff8dfa933e3a
paper_authors.name: Kai Chen
paper_authors._id: 66173f994488ff8dfa933e3b
paper_authors.name: Ping Luo
paper_title: Adapting LLaMA Decoder to Vision Transformer
paper_summary: This work examines whether decoder-only Transformers such as LLaMA, which
  were originally designed for large language models (LLMs), can be adapted to
  the computer vision field. We first "LLaMAfy" a standard ViT step-by-step to
  align with LLaMA's architecture, and find that directly applying a casual mask
  to the self-attention brings an attention collapse issue, resulting in the
  failure to the network training. We suggest to reposition the class token
  behind the image tokens with a post-sequence class token technique to overcome
  this challenge, enabling causal self-attention to efficiently capture the
  entire image's information. Additionally, we develop a soft mask strategy that
  gradually introduces a casual mask to the self-attention at the onset of
  training to facilitate the optimization behavior. The tailored model, dubbed as
  image LLaMA (iLLaMA), is akin to LLaMA in architecture and enables direct
  supervised learning. Its causal self-attention boosts computational efficiency
  and learns complex representation by elevating attention map ranks. iLLaMA
  rivals the performance with its encoder-only counterparts, achieving 75.1%
  ImageNet top-1 accuracy with only 5.7M parameters. Scaling the model to ~310M
  and pre-training on ImageNet-21K further enhances the accuracy to 86.0%.
  Extensive experiments demonstrate iLLaMA's reliable properties: calibration,
  shape-texture bias, quantization compatibility, ADE20K segmentation and CIFAR
  transfer learning. We hope our study can kindle fresh views to visual model
  design in the wave of LLMs. Pre-trained models and codes are available here.
paper_ai_keywords: decoder-only Transformers LLaMA large language models (LLMs) ViT self-attention attention collapse issue class token post-sequence class token causal mask soft mask strategy image LLaMA (iLLaMA) computational efficiency attention map ranks ImageNet top-1 accuracy pre-training ImageNet-21K calibration shape-texture bias quantization compatibility ADE20K segmentation CIFAR transfer learning