title: In-Context LoRA for Diffusion Transformers
summary: Recent research arXiv:2410.15027 has explored the use of diffusion
  transformers (DiTs) for task-agnostic image generation by simply concatenating
  attention tokens across images. However, despite substantial computational
  resources, the fidelity of the generated images remains suboptimal. In this
  study, we reevaluate and streamline this framework by hypothesizing that
  text-to-image DiTs inherently possess in-context generation capabilities,
  requiring only minimal tuning to activate them. Through diverse task
  experiments, we qualitatively demonstrate that existing text-to-image DiTs can
  effectively perform in-context generation without any tuning. Building on this
  insight, we propose a remarkably simple pipeline to leverage the in-context
  abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint
  captioning of multiple images, and (3) apply task-specific LoRA tuning using
  small datasets (e.g., 20sim 100 samples) instead of full-parameter tuning
  with large datasets. We name our models In-Context LoRA (IC-LoRA). This
  approach requires no modifications to the original DiT models, only changes to
  the training data. Remarkably, our pipeline generates high-fidelity image sets
  that better adhere to prompts. While task-specific in terms of tuning data, our
  framework remains task-agnostic in architecture and pipeline, offering a
  powerful tool for the community and providing valuable insights for further
  research on product-level task-agnostic generation systems. We release our
  code, data, and models at https://github.com/ali-vilab/In-Context-LoRA
paper_authors._id: 67246f26a2714fc915dd1104
paper_authors.name: Lianghua Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:22:06.089Z
paper_authors._id: 6522cf31777019ca30d85725
paper_authors.avatarUrl: /avatars/a180b096e438e429d445b68fe703e43f.svg
paper_authors.fullname: Lianghua Huang
paper_authors.type: user
paper_authors.user: lhhuang
paper_authors._id: 67246f26a2714fc915dd1105
paper_authors.name: Wei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T17:42:35.397Z
paper_authors._id: 62fa0ffe0697d224219a0cb7
paper_authors.avatarUrl: /avatars/f0ef59e1c0cf4ab4fe5cee08d488bd03.svg
paper_authors.fullname: Wei Wang
paper_authors.type: user
paper_authors.user: WeiWang
paper_authors._id: 67246f26a2714fc915dd1106
paper_authors.name: Zhi-Fan Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T17:42:28.526Z
paper_authors._id: 64f0337e1dccbf71c3946204
paper_authors.avatarUrl: /avatars/955d0533b4b7219773ef7b9a16c05848.svg
paper_authors.fullname: Wu Zhi-Fan
paper_authors.type: user
paper_authors.user: wuzhifan
paper_authors._id: 67246f26a2714fc915dd1107
paper_authors.name: Yupeng Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T17:42:14.408Z
paper_authors._id: 6433edd99bd5a84b5dc8f5df
paper_authors.avatarUrl: /avatars/ceaf87065ebaebabf1c0510185480ee9.svg
paper_authors.fullname: shi yupeng
paper_authors.type: user
paper_authors.user: shiyp123
paper_authors._id: 67246f26a2714fc915dd1108
paper_authors.name: Huanzhang Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T17:42:00.682Z
paper_authors._id: 64c8ad3a527d763655600f6f
paper_authors.avatarUrl: /avatars/6a4ab87fe040d666313f59d93712cf73.svg
paper_authors.fullname: Huanzhang Dou
paper_authors.type: user
paper_authors.user: Huanzhang
paper_authors._id: 67246f26a2714fc915dd1109
paper_authors.name: Chen Liang
paper_authors._id: 67246f26a2714fc915dd110a
paper_authors.name: Yutong Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T17:10:51.356Z
paper_authors._id: 64a54e468cfaa458bd6844bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
paper_authors.fullname: Yutong Feng
paper_authors.type: user
paper_authors.user: yutongfeng
paper_authors._id: 67246f26a2714fc915dd110b
paper_authors.name: Yu Liu
paper_authors._id: 67246f26a2714fc915dd110c
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T17:10:44.913Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: In-Context LoRA for Diffusion Transformers
paper_summary: Recent research arXiv:2410.15027 has explored the use of diffusion
  transformers (DiTs) for task-agnostic image generation by simply concatenating
  attention tokens across images. However, despite substantial computational
  resources, the fidelity of the generated images remains suboptimal. In this
  study, we reevaluate and streamline this framework by hypothesizing that
  text-to-image DiTs inherently possess in-context generation capabilities,
  requiring only minimal tuning to activate them. Through diverse task
  experiments, we qualitatively demonstrate that existing text-to-image DiTs can
  effectively perform in-context generation without any tuning. Building on this
  insight, we propose a remarkably simple pipeline to leverage the in-context
  abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint
  captioning of multiple images, and (3) apply task-specific LoRA tuning using
  small datasets (e.g., 20sim 100 samples) instead of full-parameter tuning
  with large datasets. We name our models In-Context LoRA (IC-LoRA). This
  approach requires no modifications to the original DiT models, only changes to
  the training data. Remarkably, our pipeline generates high-fidelity image sets
  that better adhere to prompts. While task-specific in terms of tuning data, our
  framework remains task-agnostic in architecture and pipeline, offering a
  powerful tool for the community and providing valuable insights for further
  research on product-level task-agnostic generation systems. We release our
  code, data, and models at https://github.com/ali-vilab/In-Context-LoRA
paper_ai_keywords: diffusion transformers (DiTs) in-context generation attention tokens task-agnostic image generation joint captioning LoRA tuning high-fidelity image sets