title: SEED-X: Multimodal Models with Unified Multi-granularity Comprehension
    and Generation
summary: The rapid evolution of multimodal foundation model has demonstrated
  significant progresses in vision-language understanding and generation, e.g.,
  our previous work SEED-LLaMA. However, there remains a gap between its
  capability and the real-world applicability, primarily due to the model's
  limited capacity to effectively respond to various user instructions and
  interact with diverse visual data. In this work, we focus on bridging this gap
  through integrating two enhanced features: (1) comprehending images of
  arbitrary sizes and ratios, and (2) enabling multi-granularity image
  generation. We present a unified and versatile foundation model, namely,
  SEED-X, which is able to model multi-granularity visual semantics for
  comprehension and generation tasks. Besides the competitive results on public
  benchmarks, SEED-X demonstrates its effectiveness in handling real-world
  applications across various domains after instruction tuning. We hope that our
  work will inspire future research into what can be achieved by versatile
  multimodal foundation models in real-world applications. The models, codes, and
  datasets will be released in https://github.com/AILab-CVC/SEED-X.
paper_authors._id: 662719b80c759e8f21192d3c
paper_authors.name: Yuying Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T08:17:07.027Z
paper_authors._id: 6455cc8f654d8bccae50e4d4
paper_authors.avatarUrl: /avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
paper_authors.fullname: Yuying Ge
paper_authors.type: user
paper_authors.user: tttoaster
paper_authors._id: 662719b80c759e8f21192d3d
paper_authors.name: Sijie Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T08:17:13.876Z
paper_authors._id: 652cf6bad78452c474434fa6
paper_authors.avatarUrl: /avatars/106bd7a6f8af3ae3b079a0533f6fb619.svg
paper_authors.fullname: Sijie Zhao
paper_authors.type: user
paper_authors.user: walkingshadow
paper_authors._id: 662719b80c759e8f21192d3e
paper_authors.name: Jinguo Zhu
paper_authors._id: 662719b80c759e8f21192d3f
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T08:17:26.666Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 662719b80c759e8f21192d40
paper_authors.name: Kun Yi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T08:17:34.010Z
paper_authors._id: 64071f662e309e65451fab0f
paper_authors.avatarUrl: /avatars/3c1ad16a16b82fdc2618eeeaa75fffda.svg
paper_authors.fullname: kunyi
paper_authors.type: user
paper_authors.user: Kunyi
paper_authors._id: 662719b80c759e8f21192d41
paper_authors.name: Lin Song
paper_authors._id: 662719b80c759e8f21192d42
paper_authors.name: Chen Li
paper_authors._id: 662719b80c759e8f21192d43
paper_authors.name: Xiaohan Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T08:17:59.777Z
paper_authors._id: 6564f049957ac688429a6754
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nE6q5TlyLKlJpbxCKoDN8.png
paper_authors.fullname: xiaohanding
paper_authors.type: user
paper_authors.user: DingXiaoH
paper_authors._id: 662719b80c759e8f21192d44
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T08:18:17.587Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_title: SEED-X: Multimodal Models with Unified Multi-granularity Comprehension
    and Generation
paper_summary: The rapid evolution of multimodal foundation model has demonstrated
  significant progresses in vision-language understanding and generation, e.g.,
  our previous work SEED-LLaMA. However, there remains a gap between its
  capability and the real-world applicability, primarily due to the model's
  limited capacity to effectively respond to various user instructions and
  interact with diverse visual data. In this work, we focus on bridging this gap
  through integrating two enhanced features: (1) comprehending images of
  arbitrary sizes and ratios, and (2) enabling multi-granularity image
  generation. We present a unified and versatile foundation model, namely,
  SEED-X, which is able to model multi-granularity visual semantics for
  comprehension and generation tasks. Besides the competitive results on public
  benchmarks, SEED-X demonstrates its effectiveness in handling real-world
  applications across various domains after instruction tuning. We hope that our
  work will inspire future research into what can be achieved by versatile
  multimodal foundation models in real-world applications. The models, codes, and
  datasets will be released in https://github.com/AILab-CVC/SEED-X.
paper_ai_keywords: multimodal foundation model vision-language understanding generation SEED-LLaMA user instructions visual data image comprehension multi-granularity image generation SEED-X multi-granularity visual semantics instruction tuning real-world applications public benchmarks