title: One Diffusion to Generate Them All
summary: We introduce OneDiffusion, a versatile, large-scale diffusion model that
  seamlessly supports bidirectional image synthesis and understanding across
  diverse tasks. It enables conditional generation from inputs such as text,
  depth, pose, layout, and semantic maps, while also handling tasks like image
  deblurring, upscaling, and reverse processes such as depth estimation and
  segmentation. Additionally, OneDiffusion allows for multi-view generation,
  camera pose estimation, and instant personalization using sequential image
  inputs. Our model takes a straightforward yet effective approach by treating
  all tasks as frame sequences with varying noise scales during training,
  allowing any frame to act as a conditioning image at inference time. Our
  unified training framework removes the need for specialized architectures,
  supports scalable multi-task training, and adapts smoothly to any resolution,
  enhancing both generalization and scalability. Experimental results demonstrate
  competitive performance across tasks in both generation and prediction such as
  text-to-image, multiview generation, ID preservation, depth estimation and
  camera pose estimation despite relatively small training dataset. Our code and
  checkpoint are freely available at https://github.com/lehduong/OneDiffusion
paper_authors._id: 674569ce9930041fff3f7bf0
paper_authors.name: Duong H. Le
paper_authors._id: 674569ce9930041fff3f7bf1
paper_authors.name: Tuan Pham
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:47:18.582Z
paper_authors._id: 64d3d401c1f4edb3faeae442
paper_authors.avatarUrl: /avatars/fc814ea60df1b7c552230a1de03fdd11.svg
paper_authors.fullname: Tuan Pham
paper_authors.type: user
paper_authors.user: tuanpham
paper_authors._id: 674569ce9930041fff3f7bf2
paper_authors.name: Sangho Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:46:45.705Z
paper_authors._id: 664fc02d6f16bbd9a168654d
paper_authors.avatarUrl: /avatars/9f726ede9973706bc198845464238452.svg
paper_authors.fullname: Sangho Lee
paper_authors.type: user
paper_authors.user: sanghol
paper_authors._id: 674569ce9930041fff3f7bf3
paper_authors.name: Christopher Clark
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:47:00.768Z
paper_authors._id: 6523efa8db79e53948543f79
paper_authors.avatarUrl: /avatars/11a9e82b1d6a578526e261c33ae097e3.svg
paper_authors.fullname: Christopher Clark
paper_authors.type: user
paper_authors.user: ChristopherClark123
paper_authors._id: 674569ce9930041fff3f7bf4
paper_authors.name: Aniruddha Kembhavi
paper_authors._id: 674569ce9930041fff3f7bf5
paper_authors.name: Stephan Mandt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:46:44.367Z
paper_authors._id: 6645f8a8a781869d13c40624
paper_authors.avatarUrl: /avatars/726e618765daa5e4987cc32e91994aaa.svg
paper_authors.fullname: Stephan Mandt
paper_authors.type: user
paper_authors.user: smandt
paper_authors._id: 674569ce9930041fff3f7bf6
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:46:39.290Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_authors._id: 674569ce9930041fff3f7bf7
paper_authors.name: Jiasen Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:46:33.522Z
paper_authors._id: 62b6b0397523238923221df9
paper_authors.avatarUrl: /avatars/77068771dd51df7519516cd502a88789.svg
paper_authors.fullname: Jiasenlu
paper_authors.type: user
paper_authors.user: Jiasenlu
paper_title: One Diffusion to Generate Them All
paper_summary: We introduce OneDiffusion, a versatile, large-scale diffusion model that
  seamlessly supports bidirectional image synthesis and understanding across
  diverse tasks. It enables conditional generation from inputs such as text,
  depth, pose, layout, and semantic maps, while also handling tasks like image
  deblurring, upscaling, and reverse processes such as depth estimation and
  segmentation. Additionally, OneDiffusion allows for multi-view generation,
  camera pose estimation, and instant personalization using sequential image
  inputs. Our model takes a straightforward yet effective approach by treating
  all tasks as frame sequences with varying noise scales during training,
  allowing any frame to act as a conditioning image at inference time. Our
  unified training framework removes the need for specialized architectures,
  supports scalable multi-task training, and adapts smoothly to any resolution,
  enhancing both generalization and scalability. Experimental results demonstrate
  competitive performance across tasks in both generation and prediction such as
  text-to-image, multiview generation, ID preservation, depth estimation and
  camera pose estimation despite relatively small training dataset. Our code and
  checkpoint are freely available at https://github.com/lehduong/OneDiffusion
paper_ai_keywords: diffusion model bidirectional image synthesis conditional generation depth pose layout semantic maps image deblurring upscaling depth estimation segmentation multi-view generation camera pose estimation sequential image inputs frame sequences noise scales conditioning image unified training framework multi-task training resolution adaptation text-to-image ID preservation