title: Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation
    with Multimodal Generative Pretraining
summary: We present Lumina-mGPT, a family of multimodal autoregressive models capable
  of various vision and language tasks, particularly excelling in generating
  flexible photorealistic images from text descriptions. Unlike existing
  autoregressive image generation approaches, Lumina-mGPT employs a pretrained
  decoder-only transformer as a unified framework for modeling multimodal token
  sequences. Our key insight is that a simple decoder-only transformer with
  multimodal Generative PreTraining (mGPT), utilizing the next-token prediction
  objective on massive interleaved text-image sequences, can learn broad and
  general multimodal capabilities, thereby illuminating photorealistic
  text-to-image generation. Building on these pretrained models, we propose
  Flexible Progressive Supervised Finetuning (FP-SFT) on high-quality image-text
  pairs to fully unlock their potential for high-aesthetic image synthesis at any
  resolution while maintaining their general multimodal capabilities.
  Furthermore, we introduce Ominiponent Supervised Finetuning (Omni-SFT),
  transforming Lumina-mGPT into a foundation model that seamlessly achieves
  omnipotent task unification. The resulting model demonstrates versatile
  multimodal capabilities, including visual generation tasks like flexible
  text-to-image generation and controllable generation, visual recognition tasks
  like segmentation and depth estimation, and vision-language tasks like
  multiturn visual question answering. Additionally, we analyze the differences
  and similarities between diffusion-based and autoregressive methods in a direct
  comparison.
paper_authors._id: 66b19b1cd0037f5b2d5016e7
paper_authors.name: Dongyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T15:23:53.143Z
paper_authors._id: 646f1bef075e11ca78da3bb7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg
paper_authors.fullname: Dongyang Liu (Chris Liu)
paper_authors.type: user
paper_authors.user: Cxxs
paper_authors._id: 66b19b1cd0037f5b2d5016e8
paper_authors.name: Shitian Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-06T09:59:30.180Z
paper_authors._id: 62c66504031996c36c86976a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
paper_authors.fullname: steve z
paper_authors.type: user
paper_authors.user: stzhao
paper_authors._id: 66b19b1cd0037f5b2d5016e9
paper_authors.name: Le Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T09:27:39.499Z
paper_authors._id: 6358a167f56b03ec9147074d
paper_authors.avatarUrl: /avatars/e54ea7bf0c240cf76d538296efb3976c.svg
paper_authors.fullname: Le Zhuo
paper_authors.type: user
paper_authors.user: JackyZhuo
paper_authors._id: 66b19b1cd0037f5b2d5016ea
paper_authors.name: Weifeng Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T09:28:32.532Z
paper_authors._id: 63483f3450fe0799927b9b0f
paper_authors.avatarUrl: /avatars/66c2be2d03dafcb91cec0239ca7b8b44.svg
paper_authors.fullname: Weifeng Lin
paper_authors.type: user
paper_authors.user: Singaless
paper_authors._id: 66b19b1cd0037f5b2d5016eb
paper_authors.name: Yu Qiao
paper_authors._id: 66b19b1cd0037f5b2d5016ec
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T09:28:51.119Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 66b19b1cd0037f5b2d5016ed
paper_authors.name: Peng Gao
paper_title: Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation
    with Multimodal Generative Pretraining
paper_summary: We present Lumina-mGPT, a family of multimodal autoregressive models capable
  of various vision and language tasks, particularly excelling in generating
  flexible photorealistic images from text descriptions. Unlike existing
  autoregressive image generation approaches, Lumina-mGPT employs a pretrained
  decoder-only transformer as a unified framework for modeling multimodal token
  sequences. Our key insight is that a simple decoder-only transformer with
  multimodal Generative PreTraining (mGPT), utilizing the next-token prediction
  objective on massive interleaved text-image sequences, can learn broad and
  general multimodal capabilities, thereby illuminating photorealistic
  text-to-image generation. Building on these pretrained models, we propose
  Flexible Progressive Supervised Finetuning (FP-SFT) on high-quality image-text
  pairs to fully unlock their potential for high-aesthetic image synthesis at any
  resolution while maintaining their general multimodal capabilities.
  Furthermore, we introduce Ominiponent Supervised Finetuning (Omni-SFT),
  transforming Lumina-mGPT into a foundation model that seamlessly achieves
  omnipotent task unification. The resulting model demonstrates versatile
  multimodal capabilities, including visual generation tasks like flexible
  text-to-image generation and controllable generation, visual recognition tasks
  like segmentation and depth estimation, and vision-language tasks like
  multiturn visual question answering. Additionally, we analyze the differences
  and similarities between diffusion-based and autoregressive methods in a direct
  comparison.
paper_ai_keywords: multimodal autoregressive models photorealistic images text descriptions objective-only transformer unified framework multimodal token sequences decoder-only transformer Generative PreTraining (GPT) next performance prediction massive interleaved text-image sequences multimodal capabilities photorealistic text-to-image generation Flexible Progressive Supervised Finetuning (FP-SFT) high-quality image-text pairs high-aesthetic image synthesis Ominiponent Supervised Finetuning (Omni-SFT) foundation model omnipotent task unification visual generation tasks controllable generation visual recognition tasks segmentation multimodal tasks diffusion-based methods autoregressive methods