title: VL-GPT: A Generative Pre-trained Transformer for Vision and Language
    Understanding and Generation
summary: In this work, we introduce Vision-Language Generative Pre-trained Transformer
  (VL-GPT), a transformer model proficient at concurrently perceiving and
  generating visual and linguistic data. VL-GPT achieves a unified pre-training
  approach for both image and text modalities by employing a straightforward
  auto-regressive objective, thereby enabling the model to process image and text
  as seamlessly as a language model processes text. To accomplish this, we
  initially propose a novel image tokenizer-detokenizer framework for visual
  data, specifically designed to transform raw images into a sequence of
  continuous embeddings and reconstruct them accordingly. In combination with the
  existing text tokenizer and detokenizer, this framework allows for the encoding
  of interleaved image-text data into a multimodal sequence, which can
  subsequently be fed into the transformer model. Consequently, VL-GPT can
  perform large-scale pre-training on multimodal corpora utilizing a unified
  auto-regressive objective (i.e., next-token prediction). Upon completion of
  pre-training, VL-GPT exhibits remarkable zero-shot and few-shot performance
  across a diverse range of vision and language understanding and generation
  tasks, including image captioning, visual question answering, text-to-image
  generation, and more. Additionally, the pre-trained model retrains in-context
  learning capabilities when provided with multimodal prompts. We further conduct
  instruction tuning on our VL-GPT, highlighting its exceptional potential for
  multimodal assistance. The source code and model weights shall be released.
paper_authors._id: 657bd0137e310c370218ae25
paper_authors.name: Jinguo Zhu
paper_authors._id: 657bd0137e310c370218ae26
paper_authors.name: Xiaohan Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:19:07.753Z
paper_authors._id: 6564f049957ac688429a6754
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nE6q5TlyLKlJpbxCKoDN8.png
paper_authors.fullname: xiaohanding
paper_authors.type: user
paper_authors.user: DingXiaoH
paper_authors._id: 657bd0137e310c370218ae27
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:19:16.086Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 657bd0137e310c370218ae28
paper_authors.name: Yuying Ge
paper_authors._id: 657bd0137e310c370218ae29
paper_authors.name: Sijie Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:19:25.529Z
paper_authors._id: 652cf6bad78452c474434fa6
paper_authors.avatarUrl: /avatars/106bd7a6f8af3ae3b079a0533f6fb619.svg
paper_authors.fullname: Sijie Zhao
paper_authors.type: user
paper_authors.user: walkingshadow
paper_authors._id: 657bd0137e310c370218ae2a
paper_authors.name: Hengshuang Zhao
paper_authors._id: 657bd0137e310c370218ae2b
paper_authors.name: Xiaohua Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:19:43.538Z
paper_authors._id: 642d7a9e355bb9a2a436626d
paper_authors.avatarUrl: /avatars/893c8ec0442d451cb493d2b1d0e2d5f3.svg
paper_authors.fullname: XiaoHua Wang
paper_authors.type: user
paper_authors.user: myARmy10
paper_authors._id: 657bd0137e310c370218ae2c
paper_authors.name: Ying Shan
paper_title: VL-GPT: A Generative Pre-trained Transformer for Vision and Language
    Understanding and Generation
paper_summary: In this work, we introduce Vision-Language Generative Pre-trained Transformer
  (VL-GPT), a transformer model proficient at concurrently perceiving and
  generating visual and linguistic data. VL-GPT achieves a unified pre-training
  approach for both image and text modalities by employing a straightforward
  auto-regressive objective, thereby enabling the model to process image and text
  as seamlessly as a language model processes text. To accomplish this, we
  initially propose a novel image tokenizer-detokenizer framework for visual
  data, specifically designed to transform raw images into a sequence of
  continuous embeddings and reconstruct them accordingly. In combination with the
  existing text tokenizer and detokenizer, this framework allows for the encoding
  of interleaved image-text data into a multimodal sequence, which can
  subsequently be fed into the transformer model. Consequently, VL-GPT can
  perform large-scale pre-training on multimodal corpora utilizing a unified
  auto-regressive objective (i.e., next-token prediction). Upon completion of
  pre-training, VL-GPT exhibits remarkable zero-shot and few-shot performance
  across a diverse range of vision and language understanding and generation
  tasks, including image captioning, visual question answering, text-to-image
  generation, and more. Additionally, the pre-trained model retrains in-context
  learning capabilities when provided with multimodal prompts. We further conduct
  instruction tuning on our VL-GPT, highlighting its exceptional potential for
  multimodal assistance. The source code and model weights shall be released.
paper_ai_keywords: Vision-Language Generative Pre-trained Transformer (VL-GPT) transformer model auto-regressive objective image tokenizer DeWeizer continuous embeddings multimodal sequence large-scale pre-training multimodal corpora zero-shot performance few-shot performance image captioning visual question answering text-to-image generation in-context learning instruction tuning