title: BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture,
    Training and Dataset
summary: Unifying image understanding and generation has gained growing attention in
  recent research on multimodal models. Although design choices for image
  understanding have been extensively studied, the optimal model architecture and
  training recipe for a unified framework with image generation remain
  underexplored. Motivated by the strong potential of autoregressive and
  diffusion models for high-quality generation and scalability, we conduct a
  comprehensive study of their use in unified multimodal settings, with emphasis
  on image representations, modeling objectives, and training strategies.
  Grounded in these investigations, we introduce a novel approach that employs a
  diffusion transformer to generate semantically rich CLIP image features, in
  contrast to conventional VAE-based representations. This design yields both
  higher training efficiency and improved generative quality. Furthermore, we
  demonstrate that a sequential pretraining strategy for unified models-first
  training on image understanding and subsequently on image generation-offers
  practical advantages by preserving image understanding capability while
  developing strong image generation ability. Finally, we carefully curate a
  high-quality instruction-tuning dataset BLIP3o-60k for image generation by
  prompting GPT-4o with a diverse set of captions covering various scenes,
  objects, human gestures, and more. Building on our innovative model design,
  training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art
  unified multimodal models. BLIP3-o achieves superior performance across most of
  the popular benchmarks spanning both image understanding and generation tasks.
  To facilitate future research, we fully open-source our models, including code,
  model weights, training scripts, and pretraining and instruction tuning
  datasets.
paper_authors._id: 68254419181d43c25d829239
paper_authors.name: Jiuhai Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:48:28.916Z
paper_authors._id: 6393847e3e30234ae798b7be
paper_authors.avatarUrl: /avatars/daeb8c37dff4432d837a69b87c196521.svg
paper_authors.fullname: JiuhaiChen
paper_authors.type: user
paper_authors.user: jiuhai
paper_authors._id: 68254419181d43c25d82923a
paper_authors.name: Zhiyang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:48:51.984Z
paper_authors._id: 64b6c686cf5117d7962d8f62
paper_authors.avatarUrl: /avatars/96ed7a9602aa4c21b3a3d89608e76dc8.svg
paper_authors.fullname: Zhiyang Xu
paper_authors.type: user
paper_authors.user: Zhiyang03
paper_authors._id: 68254419181d43c25d82923b
paper_authors.name: Xichen Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:31:43.038Z
paper_authors._id: 63172831c92fd6fee3181f50
paper_authors.avatarUrl: /avatars/0f57068a138cb181e9451bfc1ed3d1c0.svg
paper_authors.fullname: Xichen Pan
paper_authors.type: user
paper_authors.user: xcpan
paper_authors._id: 68254419181d43c25d82923c
paper_authors.name: Yushi Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:49:05.178Z
paper_authors._id: 62b1474bdcbad6848a91a54e
paper_authors.avatarUrl: /avatars/d7308899b46232cad4a48a0e876449a8.svg
paper_authors.fullname: Yushi Hu
paper_authors.type: user
paper_authors.user: yushihu
paper_authors._id: 68254419181d43c25d82923d
paper_authors.name: Can Qin
paper_authors._id: 68254419181d43c25d82923e
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:49:13.762Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_authors._id: 68254419181d43c25d82923f
paper_authors.name: Lifu Huang
paper_authors._id: 68254419181d43c25d829240
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:32:05.507Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_authors._id: 68254419181d43c25d829241
paper_authors.name: Saining Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:49:28.644Z
paper_authors._id: 6596422646624a86ff3b3bda
paper_authors.avatarUrl: /avatars/216e12b77e45ac5f1fa20932f5745411.svg
paper_authors.fullname: Saining Xie
paper_authors.type: user
paper_authors.user: sainx
paper_authors._id: 68254419181d43c25d829242
paper_authors.name: Silvio Savarese
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:49:36.341Z
paper_authors._id: 67d5674bbc03ef961e733ddd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/3EUXNd-mKvsXFDlr1FETh.png
paper_authors.fullname: Silvio Savarese
paper_authors.type: user
paper_authors.user: SilvioSav8
paper_authors._id: 68254419181d43c25d829243
paper_authors.name: Le Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:49:58.945Z
paper_authors._id: 63dd73e7422ca8d7f7e3698c
paper_authors.avatarUrl: /avatars/7b0f8419f6941230b81dbbbb4f273edf.svg
paper_authors.fullname: Le Xue
paper_authors.type: user
paper_authors.user: SFXX
paper_authors._id: 68254419181d43c25d829244
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:49:42.774Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_authors._id: 68254419181d43c25d829245
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:31:39.465Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_title: BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture,
    Training and Dataset
paper_summary: Unifying image understanding and generation has gained growing attention in
  recent research on multimodal models. Although design choices for image
  understanding have been extensively studied, the optimal model architecture and
  training recipe for a unified framework with image generation remain
  underexplored. Motivated by the strong potential of autoregressive and
  diffusion models for high-quality generation and scalability, we conduct a
  comprehensive study of their use in unified multimodal settings, with emphasis
  on image representations, modeling objectives, and training strategies.
  Grounded in these investigations, we introduce a novel approach that employs a
  diffusion transformer to generate semantically rich CLIP image features, in
  contrast to conventional VAE-based representations. This design yields both
  higher training efficiency and improved generative quality. Furthermore, we
  demonstrate that a sequential pretraining strategy for unified models-first
  training on image understanding and subsequently on image generation-offers
  practical advantages by preserving image understanding capability while
  developing strong image generation ability. Finally, we carefully curate a
  high-quality instruction-tuning dataset BLIP3o-60k for image generation by
  prompting GPT-4o with a diverse set of captions covering various scenes,
  objects, human gestures, and more. Building on our innovative model design,
  training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art
  unified multimodal models. BLIP3-o achieves superior performance across most of
  the popular benchmarks spanning both image understanding and generation tasks.
  To facilitate future research, we fully open-source our models, including code,
  model weights, training scripts, and pretraining and instruction tuning
  datasets.
paper_ai_summary: A diffusion transformer is used in a unified multimodal model framework to improve image generation while maintaining image understanding capabilities.
paper_ai_keywords: diffusion transformer CLIP image features VAE-based representations sequential pretraining image understanding image generation BLIP3-o instruction-tuning dataset