title: Janus: Decoupling Visual Encoding for Unified Multimodal Understanding
    and Generation
summary: In this paper, we introduce Janus, an autoregressive framework that unifies
  multimodal understanding and generation. Prior research often relies on a
  single visual encoder for both tasks, such as Chameleon. However, due to the
  differing levels of information granularity required by multimodal
  understanding and generation, this approach can lead to suboptimal performance,
  particularly in multimodal understanding. To address this issue, we decouple
  visual encoding into separate pathways, while still leveraging a single,
  unified transformer architecture for processing. The decoupling not only
  alleviates the conflict between the visual encoder's roles in understanding and
  generation, but also enhances the framework's flexibility. For instance, both
  the multimodal understanding and generation components can independently select
  their most suitable encoding methods. Experiments show that Janus surpasses
  previous unified model and matches or exceeds the performance of task-specific
  models. The simplicity, high flexibility, and effectiveness of Janus make it a
  strong candidate for next-generation unified multimodal models.
paper_authors._id: 6711d3f2b6b1901abbf8c1f3
paper_authors.name: Chengyue Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:55:10.857Z
paper_authors._id: 617526c9de8feb54b0ce45ad
paper_authors.avatarUrl: /avatars/7faf8c6f71fc318a0113d780d376c381.svg
paper_authors.fullname: Wu Chengyue
paper_authors.type: user
paper_authors.user: WuChengyue
paper_authors._id: 6711d3f2b6b1901abbf8c1f4
paper_authors.name: Xiaokang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T15:55:59.335Z
paper_authors._id: 6635e701420baf7bc3f93561
paper_authors.avatarUrl: /avatars/cdbb3085fee73ac520888977e2c575ea.svg
paper_authors.fullname: Xiaokang Chen
paper_authors.type: user
paper_authors.user: CharlesCXK
paper_authors._id: 6711d3f2b6b1901abbf8c1f5
paper_authors.name: Zhiyu Wu
paper_authors._id: 6711d3f2b6b1901abbf8c1f6
paper_authors.name: Yiyang Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T08:58:28.664Z
paper_authors._id: 667282bf992bdf3ecb3bdc86
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/aXQRm8RUJXNJYZaBQzlDu.jpeg
paper_authors.fullname: Ma_Yiyang
paper_authors.type: user
paper_authors.user: CNMaxwell
paper_authors._id: 6711d3f2b6b1901abbf8c1f7
paper_authors.name: Xingchao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T08:58:36.546Z
paper_authors._id: 646b0bbdec9a61e871799339
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646b0bbdec9a61e871799339/Xippv-ajkHkrGGAA7caLn.jpeg
paper_authors.fullname: Xingchao Liu
paper_authors.type: user
paper_authors.user: XCLiu
paper_authors._id: 6711d3f2b6b1901abbf8c1f8
paper_authors.name: Zizheng Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T08:58:43.853Z
paper_authors._id: 629d55b53a3221bb2117422c
paper_authors.avatarUrl: /avatars/e39f25cd7190d282478cdac20a7700ee.svg
paper_authors.fullname: Zizheng Pan
paper_authors.type: user
paper_authors.user: zizhpan
paper_authors._id: 6711d3f2b6b1901abbf8c1f9
paper_authors.name: Wen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:38:22.611Z
paper_authors._id: 63198e8802fb322037332f2d
paper_authors.avatarUrl: /avatars/d3f9c206e387df35beb0ed0ef1cdf865.svg
paper_authors.fullname: Wen Liu
paper_authors.type: user
paper_authors.user: doubility123
paper_authors._id: 6711d3f2b6b1901abbf8c1fa
paper_authors.name: Zhenda Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T08:59:00.682Z
paper_authors._id: 6539f6ea26df26ecd1393c37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6539f6ea26df26ecd1393c37/9VJusLLAiLhUxfBAFggpF.jpeg
paper_authors.fullname: Zhenda Xie
paper_authors.type: user
paper_authors.user: zdaxie
paper_authors._id: 6711d3f2b6b1901abbf8c1fb
paper_authors.name: Xingkai Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T08:59:15.248Z
paper_authors._id: 6475b14a37ed88a749e5e48c
paper_authors.avatarUrl: /avatars/973f4662023f0bbbc94c01dc3bb3edd3.svg
paper_authors.fullname: Xingkai Yu
paper_authors.type: user
paper_authors.user: GeeeekExplorer
paper_authors._id: 6711d3f2b6b1901abbf8c1fc
paper_authors.name: Chong Ruan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T08:59:22.321Z
paper_authors._id: 6398203609f12714ed1935c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6398203609f12714ed1935c2/uXgl0LgKnFYjq1Wz39-a6.jpeg
paper_authors.fullname: Chong Ruan
paper_authors.type: user
paper_authors.user: Chester111
paper_authors._id: 6711d3f2b6b1901abbf8c1fd
paper_authors.name: Ping Luo
paper_title: Janus: Decoupling Visual Encoding for Unified Multimodal Understanding
    and Generation
paper_summary: In this paper, we introduce Janus, an autoregressive framework that unifies
  multimodal understanding and generation. Prior research often relies on a
  single visual encoder for both tasks, such as Chameleon. However, due to the
  differing levels of information granularity required by multimodal
  understanding and generation, this approach can lead to suboptimal performance,
  particularly in multimodal understanding. To address this issue, we decouple
  visual encoding into separate pathways, while still leveraging a single,
  unified transformer architecture for processing. The decoupling not only
  alleviates the conflict between the visual encoder's roles in understanding and
  generation, but also enhances the framework's flexibility. For instance, both
  the multimodal understanding and generation components can independently select
  their most suitable encoding methods. Experiments show that Janus surpasses
  previous unified model and matches or exceeds the performance of task-specific
  models. The simplicity, high flexibility, and effectiveness of Janus make it a
  strong candidate for next-generation unified multimodal models.
paper_ai_summary: Janus, an autoregressive framework with separate visual encoding pathways within a unified transformer architecture, enhances performance in unified multimodal understanding and generation.
paper_ai_keywords: autoregressive framework multimodal understanding multimodal generation visual encoder transformer architecture encoding methods