title: Lotus: Diffusion-based Visual Foundation Model for High-quality Dense
    Prediction
summary: Leveraging the visual priors of pre-trained text-to-image diffusion models
  offers a promising solution to enhance zero-shot generalization in dense
  prediction tasks. However, existing methods often uncritically use the original
  diffusion formulation, which may not be optimal due to the fundamental
  differences between dense prediction and image generation. In this paper, we
  provide a systemic analysis of the diffusion formulation for the dense
  prediction, focusing on both quality and efficiency. And we find that the
  original parameterization type for image generation, which learns to predict
  noise, is harmful for dense prediction; the multi-step noising/denoising
  diffusion process is also unnecessary and challenging to optimize. Based on
  these insights, we introduce Lotus, a diffusion-based visual foundation model
  with a simple yet effective adaptation protocol for dense prediction.
  Specifically, Lotus is trained to directly predict annotations instead of
  noise, thereby avoiding harmful variance. We also reformulate the diffusion
  process into a single-step procedure, simplifying optimization and
  significantly boosting inference speed. Additionally, we introduce a novel
  tuning strategy called detail preserver, which achieves more accurate and
  fine-grained predictions. Without scaling up the training data or model
  capacity, Lotus achieves SoTA performance in zero-shot depth and normal
  estimation across various datasets. It also significantly enhances efficiency,
  being hundreds of times faster than most existing diffusion-based methods.
paper_authors._id: 66f62b506b0e782fa32a02db
paper_authors.name: Jing He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-03T08:31:50.539Z
paper_authors._id: 6548fae806396ffce13092f4
paper_authors.avatarUrl: /avatars/aaee62e508ed96790cab1e3f0158fafd.svg
paper_authors.fullname: Jing He
paper_authors.type: user
paper_authors.user: jingheya
paper_authors._id: 66f62b506b0e782fa32a02dc
paper_authors.name: Haodong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-27T06:58:00.560Z
paper_authors._id: 641d211e353524fe41f16387
paper_authors.avatarUrl: /avatars/c6e72c82c029b415a035beebee50b52c.svg
paper_authors.fullname: Haodong Li
paper_authors.type: user
paper_authors.user: haodongli
paper_authors._id: 66f62b506b0e782fa32a02dd
paper_authors.name: Wei Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T08:46:11.070Z
paper_authors._id: 654a2b1a83e7bfc4313a5cc7
paper_authors.avatarUrl: /avatars/dc3dfc3fcd26bb7350a9db0d075c5ea0.svg
paper_authors.fullname: Wei Yin
paper_authors.type: user
paper_authors.user: WonderingWorld
paper_authors._id: 66f62b506b0e782fa32a02de
paper_authors.name: Yixun Liang
paper_authors._id: 66f62b506b0e782fa32a02df
paper_authors.name: Leheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T07:41:30.485Z
paper_authors._id: 6343e37f73b4f9cedab1c846
paper_authors.avatarUrl: /avatars/2638af4626e8a4e3a95f845b94ad94f6.svg
paper_authors.fullname: Leheng Li
paper_authors.type: user
paper_authors.user: lilelife
paper_authors._id: 66f62b506b0e782fa32a02e0
paper_authors.name: Kaiqiang Zhou
paper_authors._id: 66f62b506b0e782fa32a02e1
paper_authors.name: Hongbo Liu
paper_authors._id: 66f62b506b0e782fa32a02e2
paper_authors.name: Bingbing Liu
paper_authors._id: 66f62b506b0e782fa32a02e3
paper_authors.name: Ying-Cong Chen
paper_title: Lotus: Diffusion-based Visual Foundation Model for High-quality Dense
    Prediction
paper_summary: Leveraging the visual priors of pre-trained text-to-image diffusion models
  offers a promising solution to enhance zero-shot generalization in dense
  prediction tasks. However, existing methods often uncritically use the original
  diffusion formulation, which may not be optimal due to the fundamental
  differences between dense prediction and image generation. In this paper, we
  provide a systemic analysis of the diffusion formulation for the dense
  prediction, focusing on both quality and efficiency. And we find that the
  original parameterization type for image generation, which learns to predict
  noise, is harmful for dense prediction; the multi-step noising/denoising
  diffusion process is also unnecessary and challenging to optimize. Based on
  these insights, we introduce Lotus, a diffusion-based visual foundation model
  with a simple yet effective adaptation protocol for dense prediction.
  Specifically, Lotus is trained to directly predict annotations instead of
  noise, thereby avoiding harmful variance. We also reformulate the diffusion
  process into a single-step procedure, simplifying optimization and
  significantly boosting inference speed. Additionally, we introduce a novel
  tuning strategy called detail preserver, which achieves more accurate and
  fine-grained predictions. Without scaling up the training data or model
  capacity, Lotus achieves SoTA performance in zero-shot depth and normal
  estimation across various datasets. It also significantly enhances efficiency,
  being hundreds of times faster than most existing diffusion-based methods.
paper_ai_summary: A diffusion-based visual foundation model, Lotus, is introduced for dense prediction tasks, achieving state-of-the-art performance in zero-shot depth and normal estimation while improving efficiency.
paper_ai_keywords: text-to-image diffusion models zero-shot generalization dense prediction diffusion formulation noise prediction multi-step noising/denoising Lotus adaptation protocol direct annotation prediction single-step diffusion process detail preserver SoTA performance