title: PUMA: Empowering Unified MLLM with Multi-granular Visual Generation
summary: Recent advancements in multimodal foundation models have yielded significant
  progress in vision-language understanding. Initial attempts have also explored
  the potential of multimodal large language models (MLLMs) for visual content
  generation. However, existing works have insufficiently addressed the varying
  granularity demands of different image generation tasks within a unified MLLM
  paradigm - from the diversity required in text-to-image generation to the
  precise controllability needed in image manipulation. In this work, we propose
  PUMA, emPowering Unified MLLM with Multi-grAnular visual generation. PUMA
  unifies multi-granular visual features as both inputs and outputs of MLLMs,
  elegantly addressing the different granularity requirements of various image
  generation tasks within a unified MLLM framework. Following multimodal
  pretraining and task-specific instruction tuning, PUMA demonstrates proficiency
  in a wide range of multimodal tasks. This work represents a significant step
  towards a truly unified MLLM capable of adapting to the granularity demands of
  various visual tasks. The code and model will be released in
  https://github.com/rongyaofang/PUMA.
paper_authors._id: 671715637892af397b9a93f3
paper_authors.name: Rongyao Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:00:53.368Z
paper_authors._id: 65b8724123d948d884b379b1
paper_authors.avatarUrl: /avatars/ce189d1d8d688c17912f9b869035b2d0.svg
paper_authors.fullname: Rongyao Fang
paper_authors.type: user
paper_authors.user: LucasFang
paper_authors._id: 671715637892af397b9a93f4
paper_authors.name: Chengqi Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:00:48.851Z
paper_authors._id: 64a2b496e2e19de17db7de65
paper_authors.avatarUrl: /avatars/241448ca487833d6cc5d57bb1fdb6ee5.svg
paper_authors.fullname: Duan Chengqi
paper_authors.type: user
paper_authors.user: gogoduan
paper_authors._id: 671715637892af397b9a93f5
paper_authors.name: Kun Wang
paper_authors._id: 671715637892af397b9a93f6
paper_authors.name: Hao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:55.990Z
paper_authors._id: 64b9033777ae61bcc80aa4f3
paper_authors.avatarUrl: /avatars/408c335395c79f3df69fd9bf70abc312.svg
paper_authors.fullname: Hao Li
paper_authors.type: user
paper_authors.user: cpsxhao
paper_authors._id: 671715637892af397b9a93f7
paper_authors.name: Hao Tian
paper_authors._id: 671715637892af397b9a93f8
paper_authors.name: Xingyu Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T08:32:52.126Z
paper_authors._id: 666d4a0fe70e5838d95aebee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/6dkjoFA_sOjCkjvcvozZ5.jpeg
paper_authors.fullname: zengxingyu
paper_authors.type: user
paper_authors.user: zengxingyu
paper_authors._id: 671715637892af397b9a93f9
paper_authors.name: Rui Zhao
paper_authors._id: 671715637892af397b9a93fa
paper_authors.name: Jifeng Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T08:33:01.448Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_authors._id: 671715637892af397b9a93fb
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T08:32:41.509Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 671715637892af397b9a93fc
paper_authors.name: Xihui Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:00:51.171Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_title: PUMA: Empowering Unified MLLM with Multi-granular Visual Generation
paper_summary: Recent advancements in multimodal foundation models have yielded significant
  progress in vision-language understanding. Initial attempts have also explored
  the potential of multimodal large language models (MLLMs) for visual content
  generation. However, existing works have insufficiently addressed the varying
  granularity demands of different image generation tasks within a unified MLLM
  paradigm - from the diversity required in text-to-image generation to the
  precise controllability needed in image manipulation. In this work, we propose
  PUMA, emPowering Unified MLLM with Multi-grAnular visual generation. PUMA
  unifies multi-granular visual features as both inputs and outputs of MLLMs,
  elegantly addressing the different granularity requirements of various image
  generation tasks within a unified MLLM framework. Following multimodal
  pretraining and task-specific instruction tuning, PUMA demonstrates proficiency
  in a wide range of multimodal tasks. This work represents a significant step
  towards a truly unified MLLM capable of adapting to the granularity demands of
  various visual tasks. The code and model will be released in
  https://github.com/rongyaofang/PUMA.
paper_ai_summary: PUMA, a unified multimodal large language model, addresses varying granularity demands in visual content generation tasks through the integration of multi-granular visual features.
paper_ai_keywords: multimodal foundation models MLLMs text-to-image generation image manipulation multi-granular visual generation multimodal pretraining task-specific instruction tuning