title: Planting a SEED of Vision in Large Language Model
summary: We present SEED, an elaborate image tokenizer that empowers Large Language
  Models (LLMs) with the emergent ability to SEE and Draw at the same time.
  Research on image tokenizers has previously reached an impasse, as frameworks
  employing quantized visual tokens have lost prominence due to subpar
  performance and convergence in multimodal comprehension (compared to BLIP-2,
  etc.) or generation (compared to Stable Diffusion, etc.). Despite the
  limitations, we remain confident in its natural capacity to unify visual and
  textual representations, facilitating scalable multimodal training with LLM's
  original recipe. In this study, we identify two crucial principles for the
  architecture and training of SEED that effectively ease subsequent alignment
  with LLMs. (1) Image tokens should be independent of 2D physical patch
  positions and instead be produced with a 1D causal dependency, exhibiting
  intrinsic interdependence that aligns with the left-to-right autoregressive
  prediction mechanism in LLMs. (2) Image tokens should capture high-level
  semantics consistent with the degree of semantic abstraction in words, and be
  optimized for both discriminativeness and reconstruction during the tokenizer
  training phase. As a result, the off-the-shelf LLM is able to perform both
  image-to-text and text-to-image generation by incorporating our SEED through
  efficient LoRA tuning. Comprehensive multimodal pretraining and instruction
  tuning, which may yield improved results, are reserved for future
  investigation. This version of SEED was trained in 5.7 days using only 64 V100
  GPUs and 5M publicly available image-text pairs. Our preliminary study
  emphasizes the great potential of discrete visual tokens in versatile
  multimodal LLMs and the importance of proper image tokenizers in broader
  research.
paper_authors._id: 64b5f88460274cbb297460c9
paper_authors.name: Yuying Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T08:09:12.877Z
paper_authors._id: 6455cc8f654d8bccae50e4d4
paper_authors.avatarUrl: /avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
paper_authors.fullname: Yuying Ge
paper_authors.type: user
paper_authors.user: tttoaster
paper_authors._id: 64b5f88460274cbb297460ca
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T08:09:32.118Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 64b5f88460274cbb297460cb
paper_authors.name: Ziyun Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T08:09:46.905Z
paper_authors._id: 633f7e0c50d83f5065d407a6
paper_authors.avatarUrl: /avatars/04d5d3334817fa9e34ade638e0155fda.svg
paper_authors.fullname: Ziyun Zeng
paper_authors.type: user
paper_authors.user: zyzeng1412
paper_authors._id: 64b5f88460274cbb297460cc
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T08:10:05.343Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 64b5f88460274cbb297460cd
paper_authors.name: Ying Shan
paper_title: Planting a SEED of Vision in Large Language Model
paper_summary: We present SEED, an elaborate image tokenizer that empowers Large Language
  Models (LLMs) with the emergent ability to SEE and Draw at the same time.
  Research on image tokenizers has previously reached an impasse, as frameworks
  employing quantized visual tokens have lost prominence due to subpar
  performance and convergence in multimodal comprehension (compared to BLIP-2,
  etc.) or generation (compared to Stable Diffusion, etc.). Despite the
  limitations, we remain confident in its natural capacity to unify visual and
  textual representations, facilitating scalable multimodal training with LLM's
  original recipe. In this study, we identify two crucial principles for the
  architecture and training of SEED that effectively ease subsequent alignment
  with LLMs. (1) Image tokens should be independent of 2D physical patch
  positions and instead be produced with a 1D causal dependency, exhibiting
  intrinsic interdependence that aligns with the left-to-right autoregressive
  prediction mechanism in LLMs. (2) Image tokens should capture high-level
  semantics consistent with the degree of semantic abstraction in words, and be
  optimized for both discriminativeness and reconstruction during the tokenizer
  training phase. As a result, the off-the-shelf LLM is able to perform both
  image-to-text and text-to-image generation by incorporating our SEED through
  efficient LoRA tuning. Comprehensive multimodal pretraining and instruction
  tuning, which may yield improved results, are reserved for future
  investigation. This version of SEED was trained in 5.7 days using only 64 V100
  GPUs and 5M publicly available image-text pairs. Our preliminary study
  emphasizes the great potential of discrete visual tokens in versatile
  multimodal LLMs and the importance of proper image tokenizers in broader
  research.
paper_ai_keywords: image tokenizer Large Language Models (LLMs) multimodal comprehension quantized visual tokens multimodal training image tokens 1D causal dependency left-to-right autoregressive prediction mechanism high-level semantics discriminativeness reconstruction LoRA tuning comprehensive multimodal pretraining instruction tuning V100 GPUs image-to-text generation text-to-image generation discrete visual tokens versatile multimodal LLMs