VisionLLaMA: A Unified LLaMA Interface for Vision Tasks
Large language models are built on top of a transformer-based architecture to
process textual inputs. For example, the LLaMA stands out among many
open-source implementations. Can the same transformer be used to process 2D
images? In this paper, we answer this question by unveiling a LLaMA-like vision
transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored
for this purpose. VisionLLaMA is a unified and generic modelling framework for
solving most vision tasks. We extensively evaluate its effectiveness using
typical pre-training paradigms in a good portion of downstream tasks of image
perception and especially image generation. In many cases, VisionLLaMA have
exhibited substantial gains over the previous state-of-the-art vision
transformers. We believe that VisionLLaMA can serve as a strong new baseline
model for vision generation and understanding. Our code will be released at
https://github.com/Meituan-AutoML/VisionLLaMA.
65e55aec521cc7807343e364
Xiangxiang Chu
65e55aec521cc7807343e365
Jianlin Su
admin_assigned
2024-03-04T10:23:14.680Z
6404982cad54665351d7c1e0
/avatars/8fb6d01802cbd4a1cbb7f6a0d83faa3a.svg
jianlin su
user
bojone
65e55aec521cc7807343e366
Bo Zhang
claimed_verified
2024-03-05T08:49:22.910Z
64688d93dcbb937d56b4fbae
https://cdn-avatars.huggingface.co/v1/production/uploads/64688d93dcbb937d56b4fbae/AHf01DkvhL8p_E9ldwggW.jpeg
Bo Zhang
user
boydcheung
65e55aec521cc7807343e367
Chunhua Shen
VisionLLaMA: A Unified LLaMA Interface for Vision Tasks
Large language models are built on top of a transformer-based architecture to
process textual inputs. For example, the LLaMA stands out among many
open-source implementations. Can the same transformer be used to process 2D
images? In this paper, we answer this question by unveiling a LLaMA-like vision
transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored
for this purpose. VisionLLaMA is a unified and generic modelling framework for
solving most vision tasks. We extensively evaluate its effectiveness using
typical pre-training paradigms in a good portion of downstream tasks of image
perception and especially image generation. In many cases, VisionLLaMA have
exhibited substantial gains over the previous state-of-the-art vision
transformers. We believe that VisionLLaMA can serve as a strong new baseline
model for vision generation and understanding. Our code will be released at
https://github.com/Meituan-AutoML/VisionLLaMA.
A LLaMA-like vision transformer, VisionLLaMA, is introduced for processing 2D images and achieves state-of-the-art performance in various vision tasks.
transformer-based architecture VisionLLaMA pre-training paradigms downstream tasks image perception image generation vision transformers