PUMA: Empowering Unified MLLM with Multi-granular Visual Generation
Recent advancements in multimodal foundation models have yielded significant
progress in vision-language understanding. Initial attempts have also explored
the potential of multimodal large language models (MLLMs) for visual content
generation. However, existing works have insufficiently addressed the varying
granularity demands of different image generation tasks within a unified MLLM
paradigm - from the diversity required in text-to-image generation to the
precise controllability needed in image manipulation. In this work, we propose
PUMA, emPowering Unified MLLM with Multi-grAnular visual generation. PUMA
unifies multi-granular visual features as both inputs and outputs of MLLMs,
elegantly addressing the different granularity requirements of various image
generation tasks within a unified MLLM framework. Following multimodal
pretraining and task-specific instruction tuning, PUMA demonstrates proficiency
in a wide range of multimodal tasks. This work represents a significant step
towards a truly unified MLLM capable of adapting to the granularity demands of
various visual tasks. The code and model will be released in
https://github.com/rongyaofang/PUMA.
671715637892af397b9a93f3
Rongyao Fang
claimed_verified
2024-10-22T08:00:53.368Z
65b8724123d948d884b379b1
/avatars/ce189d1d8d688c17912f9b869035b2d0.svg
Rongyao Fang
user
LucasFang
671715637892af397b9a93f4
Chengqi Duan
claimed_verified
2024-10-22T08:00:48.851Z
64a2b496e2e19de17db7de65
/avatars/241448ca487833d6cc5d57bb1fdb6ee5.svg
Duan Chengqi
user
gogoduan
671715637892af397b9a93f5
Kun Wang
671715637892af397b9a93f6
Hao Li
claimed_verified
2024-12-16T09:42:55.990Z
64b9033777ae61bcc80aa4f3
/avatars/408c335395c79f3df69fd9bf70abc312.svg
Hao Li
user
cpsxhao
671715637892af397b9a93f7
Hao Tian
671715637892af397b9a93f8
Xingyu Zeng
admin_assigned
2024-10-22T08:32:52.126Z
666d4a0fe70e5838d95aebee
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/6dkjoFA_sOjCkjvcvozZ5.jpeg
zengxingyu
user
zengxingyu
671715637892af397b9a93f9
Rui Zhao
671715637892af397b9a93fa
Jifeng Dai
admin_assigned
2024-10-22T08:33:01.448Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
671715637892af397b9a93fb
Hongsheng Li
admin_assigned
2024-10-22T08:32:41.509Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
671715637892af397b9a93fc
Xihui Liu
claimed_verified
2024-10-22T08:00:51.171Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
PUMA: Empowering Unified MLLM with Multi-granular Visual Generation
Recent advancements in multimodal foundation models have yielded significant
progress in vision-language understanding. Initial attempts have also explored
the potential of multimodal large language models (MLLMs) for visual content
generation. However, existing works have insufficiently addressed the varying
granularity demands of different image generation tasks within a unified MLLM
paradigm - from the diversity required in text-to-image generation to the
precise controllability needed in image manipulation. In this work, we propose
PUMA, emPowering Unified MLLM with Multi-grAnular visual generation. PUMA
unifies multi-granular visual features as both inputs and outputs of MLLMs,
elegantly addressing the different granularity requirements of various image
generation tasks within a unified MLLM framework. Following multimodal
pretraining and task-specific instruction tuning, PUMA demonstrates proficiency
in a wide range of multimodal tasks. This work represents a significant step
towards a truly unified MLLM capable of adapting to the granularity demands of
various visual tasks. The code and model will be released in
https://github.com/rongyaofang/PUMA.
PUMA, a unified multimodal large language model, addresses varying granularity demands in visual content generation tasks through the integration of multi-granular visual features.
multimodal foundation models MLLMs text-to-image generation image manipulation multi-granular visual generation multimodal pretraining task-specific instruction tuning