title: EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via
    Multimodal LLM
summary: Significant achievements in personalization of diffusion models have been
  witnessed. Conventional tuning-free methods mostly encode multiple reference
  images by averaging their image embeddings as the injection condition, but such
  an image-independent operation cannot perform interaction among images to
  capture consistent visual elements within multiple references. Although the
  tuning-based Low-Rank Adaptation (LoRA) can effectively extract consistent
  elements within multiple images through the training process, it necessitates
  specific finetuning for each distinct image group. This paper introduces
  EasyRef, a novel plug-and-play adaptation method that enables diffusion models
  to be conditioned on multiple reference images and the text prompt. To
  effectively exploit consistent visual elements within multiple images, we
  leverage the multi-image comprehension and instruction-following capabilities
  of the multimodal large language model (MLLM), prompting it to capture
  consistent visual elements based on the instruction. Besides, injecting the
  MLLM's representations into the diffusion process through adapters can easily
  generalize to unseen domains, mining the consistent visual elements within
  unseen data. To mitigate computational costs and enhance fine-grained detail
  preservation, we introduce an efficient reference aggregation strategy and a
  progressive training scheme. Finally, we introduce MRBench, a new
  multi-reference image generation benchmark. Experimental results demonstrate
  EasyRef surpasses both tuning-free methods like IP-Adapter and tuning-based
  methods like LoRA, achieving superior aesthetic quality and robust zero-shot
  generalization across diverse domains.
paper_authors._id: 675bad005b45ed6ecd315666
paper_authors.name: Zhuofan Zong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:15.087Z
paper_authors._id: 64db05f23725f8d9a9f566c4
paper_authors.avatarUrl: /avatars/345b6d2dfadb91ac1f02be023dd27c8f.svg
paper_authors.fullname: Zhuofan Zong
paper_authors.type: user
paper_authors.user: zongzhuofan
paper_authors._id: 675bad005b45ed6ecd315667
paper_authors.name: Dongzhi Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:14.337Z
paper_authors._id: 6349214f8146350b3a4c5cdf
paper_authors.avatarUrl: /avatars/cfd24caac9a87efb528d0f4c375932bc.svg
paper_authors.fullname: Dongzhi Jiang
paper_authors.type: user
paper_authors.user: CaraJ
paper_authors._id: 675bad005b45ed6ecd315668
paper_authors.name: Bingqi Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:20:31.312Z
paper_authors._id: 64424d1d5933b4560698ff86
paper_authors.avatarUrl: /avatars/f44d73744796151eda0ef0035ad10a2b.svg
paper_authors.fullname: bingqi ma
paper_authors.type: user
paper_authors.user: A157801
paper_authors._id: 675bad005b45ed6ecd315669
paper_authors.name: Guanglu Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:20:19.713Z
paper_authors._id: 636a2efffbc31ee68fee5c1c
paper_authors.avatarUrl: /avatars/21e119f23e1e6fe926567409e74d1a89.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: Guanglu
paper_authors._id: 675bad005b45ed6ecd31566a
paper_authors.name: Hao Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:20:07.328Z
paper_authors._id: 651bda4337fecec1fe8bb09f
paper_authors.avatarUrl: /avatars/f1b3d2b9874022e64d8e5dfeac0e7120.svg
paper_authors.fullname: Hao Shao
paper_authors.type: user
paper_authors.user: deepcs233
paper_authors._id: 675bad005b45ed6ecd31566b
paper_authors.name: Dazhong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:20:00.987Z
paper_authors._id: 645322c20446280667d6fce8
paper_authors.avatarUrl: /avatars/776cf496f58f67bc2c80d9c6e5ff433e.svg
paper_authors.fullname: Dazhong Sehn
paper_authors.type: user
paper_authors.user: Dazhong
paper_authors._id: 675bad005b45ed6ecd31566c
paper_authors.name: Yu Liu
paper_authors._id: 675bad005b45ed6ecd31566d
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:19:33.809Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via
    Multimodal LLM
paper_summary: Significant achievements in personalization of diffusion models have been
  witnessed. Conventional tuning-free methods mostly encode multiple reference
  images by averaging their image embeddings as the injection condition, but such
  an image-independent operation cannot perform interaction among images to
  capture consistent visual elements within multiple references. Although the
  tuning-based Low-Rank Adaptation (LoRA) can effectively extract consistent
  elements within multiple images through the training process, it necessitates
  specific finetuning for each distinct image group. This paper introduces
  EasyRef, a novel plug-and-play adaptation method that enables diffusion models
  to be conditioned on multiple reference images and the text prompt. To
  effectively exploit consistent visual elements within multiple images, we
  leverage the multi-image comprehension and instruction-following capabilities
  of the multimodal large language model (MLLM), prompting it to capture
  consistent visual elements based on the instruction. Besides, injecting the
  MLLM's representations into the diffusion process through adapters can easily
  generalize to unseen domains, mining the consistent visual elements within
  unseen data. To mitigate computational costs and enhance fine-grained detail
  preservation, we introduce an efficient reference aggregation strategy and a
  progressive training scheme. Finally, we introduce MRBench, a new
  multi-reference image generation benchmark. Experimental results demonstrate
  EasyRef surpasses both tuning-free methods like IP-Adapter and tuning-based
  methods like LoRA, achieving superior aesthetic quality and robust zero-shot
  generalization across diverse domains.
paper_ai_summary: EasyRef enhances diffusion models for multi-reference image generation by leveraging multimodal large language models and efficient training strategies, outperforming existing methods in aesthetic quality and generalization.
paper_ai_keywords: diffusion models EasyRef Low-Rank Adaptation (LoRA) multimodal large language model (MLLM) reference aggregation progressive training MRBench IP-Adapter