title: VisionLLaMA: A Unified LLaMA Interface for Vision Tasks
summary: Large language models are built on top of a transformer-based architecture to
  process textual inputs. For example, the LLaMA stands out among many
  open-source implementations. Can the same transformer be used to process 2D
  images? In this paper, we answer this question by unveiling a LLaMA-like vision
  transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored
  for this purpose. VisionLLaMA is a unified and generic modelling framework for
  solving most vision tasks. We extensively evaluate its effectiveness using
  typical pre-training paradigms in a good portion of downstream tasks of image
  perception and especially image generation. In many cases, VisionLLaMA have
  exhibited substantial gains over the previous state-of-the-art vision
  transformers. We believe that VisionLLaMA can serve as a strong new baseline
  model for vision generation and understanding. Our code will be released at
  https://github.com/Meituan-AutoML/VisionLLaMA.
paper_authors._id: 65e55aec521cc7807343e364
paper_authors.name: Xiangxiang Chu
paper_authors._id: 65e55aec521cc7807343e365
paper_authors.name: Jianlin Su
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-04T10:23:14.680Z
paper_authors._id: 6404982cad54665351d7c1e0
paper_authors.avatarUrl: /avatars/8fb6d01802cbd4a1cbb7f6a0d83faa3a.svg
paper_authors.fullname: jianlin su
paper_authors.type: user
paper_authors.user: bojone
paper_authors._id: 65e55aec521cc7807343e366
paper_authors.name: Bo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-05T08:49:22.910Z
paper_authors._id: 64688d93dcbb937d56b4fbae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64688d93dcbb937d56b4fbae/AHf01DkvhL8p_E9ldwggW.jpeg
paper_authors.fullname: Bo Zhang
paper_authors.type: user
paper_authors.user: boydcheung
paper_authors._id: 65e55aec521cc7807343e367
paper_authors.name: Chunhua Shen
paper_title: VisionLLaMA: A Unified LLaMA Interface for Vision Tasks
paper_summary: Large language models are built on top of a transformer-based architecture to
  process textual inputs. For example, the LLaMA stands out among many
  open-source implementations. Can the same transformer be used to process 2D
  images? In this paper, we answer this question by unveiling a LLaMA-like vision
  transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored
  for this purpose. VisionLLaMA is a unified and generic modelling framework for
  solving most vision tasks. We extensively evaluate its effectiveness using
  typical pre-training paradigms in a good portion of downstream tasks of image
  perception and especially image generation. In many cases, VisionLLaMA have
  exhibited substantial gains over the previous state-of-the-art vision
  transformers. We believe that VisionLLaMA can serve as a strong new baseline
  model for vision generation and understanding. Our code will be released at
  https://github.com/Meituan-AutoML/VisionLLaMA.
paper_ai_keywords: transformer-based architecture VisionLLaMA unified and generic modelling framework pre-training paradigms image perception image generation state-of-the-art vision transformers baseline model vision generation vision understanding